ELK Stack生產實踐——Python操作Elasticsearch

作者：崔亮的博客 2023-09-05 07:40:37

我們除了可以使用kibana頁面管理操作ElasticSearch外，elasticsearch還支持通過SDK和RESTful API接口進行操作。通過sdk我們可以非常便捷的將我們的項目與elasticsearch對接，實現數據的增刪改查。通過API接口我們可以對elasticsearch集群進行一些例如索引調整、集群故障處理等進階操作。

安裝與連接

安裝

使用pip安裝Elasticsearch的Python 客戶端，pip包版本要與ES版本保持一致。

pip install elasticsearch==8.8.2

elasticsearch支持多種證書和身份認證方式，接下來演示幾種常用連接認證方式。

使用CA證書+Basic Auth連接

指定CA證書路徑為certs/http_ca.crt，代碼如下所示：

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka'),
        )

    def info(self):
        logger.info(self.client.info())

    def disconnect(self):
        self.client.close()

        
if __name__ == '__main__':
    es = ES()
    es.info()
    es.disconnect()

點擊執行后，控制臺成功打印了當前es的信息

2023-08-07 16:08:31.743 | INFO     | __main__:info:14 - {'name': 'f70457f06c3c', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'JhbjrKJzTVqXlkpHWDFhFg', 'version': {'number': '8.8.2', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '98e1271edf932a480e4262a471281f1ee295ce6b', 'build_date': '2023-06-26T05:16:16.196344851Z', 'build_snapshot': False, 'lucene_version': '9.6.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

使用證書指紋+Basic Auth連接

生成證書指紋

(venv) ?  pythonProject openssl x509 -fingerprint -sha256 -noout -in certs/http_ca.crt
sha256 Fingerprint=B2:DD:C4:5B:F1:C9:9B:B0:4E:6C:63:A9:FF:80:EC:CF:03:EF:66:80:12:94:9B:70:98:74:B6:93:BA:03:BC:DB

指定使用證書指紋連接ES，代碼如下所示：

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ssl_assert_fingerprint="B2:DD:C4:5B:F1:C9:9B:B0:4E:6C:63:A9:FF:80:EC:CF:03:EF:66:80:12:94:9B:70:98:74:B6:93:BA:03:BC:DB",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka'),
        )

    def info(self):
        logger.info(self.client.info())

    def disconnect(self):
        self.client.close()


if __name__ == '__main__':
    es = ES()
    es.info()
    es.disconnect()

使用CA證書+token連接

使用devtools工具，在enterprise-search-server服務賬號下創建一個名為python-client的額令牌，發送如下請求：

# 請求
POST /_security/service/elastic/enterprise-search-server/credential/token/python-client
# 響應
{
  "created": true,
  "token": {
    "name": "python-client",
    "value": "AAEAAWVsYXN0aWMvZW50ZXJwcmlzZS1zZWFyY2gtc2VydmVyL3B5dGhvbi1jbGllbnQ6V2hMRDBVa2lRbWl0dWVicnJhY21WUQ"
  }
}

指定認證方式為token認證，并使用CA證書，代碼如下：

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            bearer_auth="AAEAAWVsYXN0aWMvZW50ZXJwcmlzZS1zZWFyY2gtc2VydmVyL3B5dGhvbi1jbGllbnQ6V2hMRDBVa2lRbWl0dWVicnJhY21WUQ"
        )

    def info(self):
        logger.info(self.client.info())

    def disconnect(self):
        self.client.close()


if __name__ == '__main__':
    es = ES()
    es.info()
    es.disconnect()

使用CA證書+key連接

在kibana的API密鑰界面，點擊創建API密鑰。

圖片

創建完成后，點擊復制密鑰內容。

圖片

指定認證方式為api_key認證，并使用CA證書，代碼如下：

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            api_key="RXEwNHo0a0I2RlZiUXBkYUt0UkI6RFBFc2NBbG9RZ0tmRzZabVZkSjI4UQ=="
        )

    def info(self):
        logger.info(self.client.info())

    def disconnect(self):
        self.client.close()


if __name__ == '__main__':
    es = ES()
    es.info()
    es.disconnect()

客戶端基礎操作

查詢結果統計

例如查詢指定時間范圍內access_status值為502、503、504的記錄條數。

先在devtools調試查詢語句，查詢語句如下

GET logs-myapp-default/_count
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "access_status": [
              "502",
              "503",
              "504"
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2023-08-07T17:20:00.000+08:00",
              "lt": "2023-08-07T17:30:00.000+08:00"
            }
          }
        }
      ]
    }
  }
}

代碼如下

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def query_count(self, index: str, query: dict) -> None:
        """
        DSL查詢結果統計
        :param index:查詢索引
        :param query: 查詢語句
        :return: 查詢結果統計
        """
        result = self.client.count(index=index, query=query)
        logger.info(result)


if __name__ == '__main__':
    es = ES()
    query = {"bool": {"filter": [{"terms": {"access_status": ["502", "503", "504"]}}, {
        "range": {"@timestamp": {"gte": "2023-08-07T17:20:00.000+08:00", "lt": "2023-08-07T17:30:00.000+08:00"}}}]}}
    es.query_count('logs-myapp-default', query)

運行結果如下，從結果字段可知，符合查詢條件的文檔總計173條。

2023-08-07 17:57:47.533 | INFO     | __main__:query_count:21 - {'count': 173, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}}

DSL查詢

Python客戶端同樣也支持靈活的DSL查詢，與7X版本的sdk區別在于，之前版本的查詢語句直接傳入body參數即可，在8X版本需要分別傳入query、sort、size等參數。

例如查詢指定時間范圍內access_status值為502、503、504的記錄條數，并按@timestamp排序，返回前20條記錄。

查詢語句如下

GET logs-myapp-default/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "access_status": [
              "502",
              "503",
              "504"
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2023-08-07T17:20:00.000+08:00",
              "lt": "2023-08-07T17:30:00.000+08:00"
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ],
  "size": 20
}

代碼如下

from elasticsearch import Elasticsearch
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def query_dsl(self, index: str, query: dict, sort: list, size: int) -> None:
        """
        獲取DSL查詢結果的內容
        :param index:查詢索引
        :param query:查詢語句
        :param sort:排序參數
        :param size:分頁參數
        :return:
        """
        result = self.client.search(index=index, query=query, sort=sort, size=size)
        logger.info(result)


if __name__ == '__main__':
    es = ES()
    query = {"bool": {"filter": [{"terms": {"access_status": ["502", "503", "504"]}}, {
        "range": {"@timestamp": {"gte": "2023-08-07T17:20:00.000+08:00", "lt": "2023-08-07T17:30:00.000+08:00"}}}]}}
    sort = [{"@timestamp": {"order": "asc"}}]
    es.query_dsl('logs-myapp-default', query, sort, 20)

運行結果如下

2023-08-07 18:09:43.781 | INFO     | __main__:query_dsl:23 - {'took': 15, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 173, 'relation': 'eq'}, 'max_score': None, 'hits': [{'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': '5K5Lz4kB6FVbQpda6wfv', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:04.706', 'server_name': 'cu-7.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '148.158.17.189', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:20:03.852+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'POST', 'remote_address': '148.158.17.189', 'message': "2023-08-07 17:20:04.706 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'POST', 'request_uri': '/account/', 'request_length': 89, 'remote_address': '148.158.17.189', 'server_name': 'cu-7.cn', 'time_start': '2023-08-07T17:20:03.852+08:00', 'time_finish': '2023-08-07T17:20:04.961+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:05.532Z', 'ecs': {'version': '8.0.0'}, 'request_length': 89, 'time_finish': '2023-08-07T17:20:04.961+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400005532]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'A65Lz4kB6FVbQpda-wie', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:09.377', 'server_name': 'cu-6.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '149.136.192.44', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:20:08.499+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '149.136.192.44', 'message': "2023-08-07 17:20:09.377 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/management/', 'request_length': 64, 'remote_address': '149.136.192.44', 'server_name': 'cu-6.cn', 'time_start': '2023-08-07T17:20:08.499+08:00', 'time_finish': '2023-08-07T17:20:09.780+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36'}", 'request_uri': '/management/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:09.542Z', 'ecs': {'version': '8.0.0'}, 'request_length': 64, 'time_finish': '2023-08-07T17:20:09.780+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400009542]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'DK5Mz4kB6FVbQpdaBwhh', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:12.298', 'server_name': 'cu-1.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '173.36.186.69', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:20:11.384+08:00', 'level': 'error', 'access_status': 503, 'request_method': 'PUT', 'remote_address': '173.36.186.69', 'message': "2023-08-07 17:20:12.298 | ERROR    | __main__:debug_log:46 - {'access_status': 503, 'request_method': 'PUT', 'request_uri': '/public/', 'request_length': 84, 'remote_address': '173.36.186.69', 'server_name': 'cu-1.cn', 'time_start': '2023-08-07T17:20:11.384+08:00', 'time_finish': '2023-08-07T17:20:12.385+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:12.554Z', 'ecs': {'version': '8.0.0'}, 'request_length': 84, 'time_finish': '2023-08-07T17:20:12.385+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400012554]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Da5Mz4kB6FVbQpdaBwhh', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:12.491', 'server_name': 'cu-3.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '173.159.200.4', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:20:12.470+08:00', 'level': 'error', 'access_status': 502, 'request_method': 'GET', 'remote_address': '173.159.200.4', 'message': "2023-08-07 17:20:12.491 | ERROR    | __main__:debug_log:46 - {'access_status': 502, 'request_method': 'GET', 'request_uri': '/account/', 'request_length': 52, 'remote_address': '173.159.200.4', 'server_name': 'cu-3.cn', 'time_start': '2023-08-07T17:20:12.470+08:00', 'time_finish': '2023-08-07T17:20:13.423+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:12.555Z', 'ecs': {'version': '8.0.0'}, 'request_length': 52, 'time_finish': '2023-08-07T17:20:13.423+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400012555]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Za5Mz4kB6FVbQpdaNgiZ', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:23.636', 'server_name': 'cm-1.cn', 'geoip': {'timezone': 'America/Chicago', 'latitude': 29.7009, 'ip': '129.106.215.14', 'continent_code': 'NA', 'city_name': 'Houston', 'country_code2': 'US', 'dma_code': 618, 'country_name': 'United States', 'country_code3': 'US', 'location': {'lon': -95.5885, 'lat': 29.7009}, 'region_name': 'Texas', 'postal_code': '77072', 'longitude': -95.5885, 'region_code': 'TX'}, 'time_start': '2023-08-07T17:20:23.511+08:00', 'level': 'error', 'access_status': 503, 'request_method': 'GET', 'remote_address': '129.106.215.14', 'message': "2023-08-07 17:20:23.636 | ERROR    | __main__:debug_log:46 - {'access_status': 503, 'request_method': 'GET', 'request_uri': '/login/', 'request_length': 21, 'remote_address': '129.106.215.14', 'server_name': 'cm-1.cn', 'time_start': '2023-08-07T17:20:23.511+08:00', 'time_finish': '2023-08-07T17:20:24.535+08:00', 'http_user_agent': 'Safari 11\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5'}", 'request_uri': '/login/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Safari 11\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:24.582Z', 'ecs': {'version': '8.0.0'}, 'request_length': 21, 'time_finish': '2023-08-07T17:20:24.535+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400024582]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'ja5Mz4kB6FVbQpdaRggO', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:28.093', 'server_name': 'cu-6.cn', 'geoip': {'timezone': 'Australia/Sydney', 'ip': '143.238.52.149', 'latitude': -33.494, 'country_name': 'Australia', 'country_code2': 'AU', 'country_code3': 'AU', 'continent_code': 'OC', 'location': {'lon': 143.2104, 'lat': -33.494}, 'longitude': 143.2104}, 'time_start': '2023-08-07T17:20:27.507+08:00', 'level': 'error', 'access_status': 502, 'request_method': 'POST', 'remote_address': '143.238.52.149', 'message': "2023-08-07 17:20:28.093 | ERROR    | __main__:debug_log:46 - {'access_status': 502, 'request_method': 'POST', 'request_uri': '/public/', 'request_length': 51, 'remote_address': '143.238.52.149', 'server_name': 'cu-6.cn', 'time_start': '2023-08-07T17:20:27.507+08:00', 'time_finish': '2023-08-07T17:20:28.906+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:28.592Z', 'ecs': {'version': '8.0.0'}, 'request_length': 51, 'time_finish': '2023-08-07T17:20:28.906+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400028592]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'q65Mz4kB6FVbQpdaTQje', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:30.567', 'server_name': 'cu-5.cn', 'geoip': {}, 'time_start': '2023-08-07T17:20:30.142+08:00', 'level': 'error', 'access_status': 503, 'request_method': 'GET', 'remote_address': '127.0.208.73', 'message': "2023-08-07 17:20:30.567 | ERROR    | __main__:debug_log:46 - {'access_status': 503, 'request_method': 'GET', 'request_uri': '/account/', 'request_length': 80, 'remote_address': '127.0.208.73', 'server_name': 'cu-5.cn', 'time_start': '2023-08-07T17:20:30.142+08:00', 'time_finish': '2023-08-07T17:20:31.493+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied', '_geoip_lookup_failure'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:30.597Z', 'ecs': {'version': '8.0.0'}, 'request_length': 80, 'time_finish': '2023-08-07T17:20:31.493+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400030597]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'rK5Mz4kB6FVbQpdaUQjF', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:30.877', 'server_name': 'cu-8.cn', 'geoip': {'timezone': 'Australia/Sydney', 'latitude': -33.8715, 'ip': '120.150.186.169', 'continent_code': 'OC', 'city_name': 'Sydney', 'country_code2': 'AU', 'country_name': 'Australia', 'country_code3': 'AU', 'location': {'lon': 151.2006, 'lat': -33.8715}, 'region_name': 'New South Wales', 'postal_code': '2000', 'longitude': 151.2006, 'region_code': 'NSW'}, 'time_start': '2023-08-07T17:20:30.545+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'DELETE', 'remote_address': '120.150.186.169', 'message': "2023-08-07 17:20:30.877 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'DELETE', 'request_uri': '/login/', 'request_length': 56, 'remote_address': '120.150.186.169', 'server_name': 'cu-8.cn', 'time_start': '2023-08-07T17:20:30.545+08:00', 'time_finish': '2023-08-07T17:20:31.057+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33'}", 'request_uri': '/login/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:31.599Z', 'ecs': {'version': '8.0.0'}, 'request_length': 56, 'time_finish': '2023-08-07T17:20:31.057+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400031599]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Jq5Mz4kB6FVbQpdadAn5', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:40.236', 'server_name': 'cu-4.cn', 'geoip': {'timezone': 'Asia/Shanghai', 'ip': '120.53.138.78', 'latitude': 34.7732, 'country_name': 'China', 'country_code2': 'CN', 'country_code3': 'CN', 'continent_code': 'AS', 'location': {'lon': 113.722, 'lat': 34.7732}, 'longitude': 113.722}, 'time_start': '2023-08-07T17:20:40.030+08:00', 'level': 'error', 'access_status': 502, 'request_method': 'GET', 'remote_address': '120.53.138.78', 'message': "2023-08-07 17:20:40.236 | ERROR    | __main__:debug_log:46 - {'access_status': 502, 'request_method': 'GET', 'request_uri': '/account/', 'request_length': 9, 'remote_address': '120.53.138.78', 'server_name': 'cu-4.cn', 'time_start': '2023-08-07T17:20:40.030+08:00', 'time_finish': '2023-08-07T17:20:40.434+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:40.613Z', 'ecs': {'version': '8.0.0'}, 'request_length': 9, 'time_finish': '2023-08-07T17:20:40.434+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400040613]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'WK5Mz4kB6FVbQpdajAmI', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:46.116', 'server_name': 'cu-3.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '209.216.41.21', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:20:45.621+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '209.216.41.21', 'message': "2023-08-07 17:20:46.116 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/account/', 'request_length': 19, 'remote_address': '209.216.41.21', 'server_name': 'cu-3.cn', 'time_start': '2023-08-07T17:20:45.621+08:00', 'time_finish': '2023-08-07T17:20:47.051+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:46.640Z', 'ecs': {'version': '8.0.0'}, 'request_length': 19, 'time_finish': '2023-08-07T17:20:47.051+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400046640]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'g65Mz4kB6FVbQpdanAkv', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:50.607', 'server_name': 'cu-6.cn', 'geoip': {'timezone': 'Asia/Tokyo', 'ip': '106.187.9.109', 'latitude': 35.6897, 'country_name': 'Japan', 'country_code2': 'JP', 'country_code3': 'JP', 'continent_code': 'AS', 'location': {'lon': 139.6895, 'lat': 35.6897}, 'longitude': 139.6895}, 'time_start': '2023-08-07T17:20:49.888+08:00', 'level': 'error', 'access_status': 502, 'request_method': 'GET', 'remote_address': '106.187.9.109', 'message': "2023-08-07 17:20:50.607 | ERROR    | __main__:debug_log:46 - {'access_status': 502, 'request_method': 'GET', 'request_uri': '/public/', 'request_length': 26, 'remote_address': '106.187.9.109', 'server_name': 'cu-6.cn', 'time_start': '2023-08-07T17:20:49.888+08:00', 'time_finish': '2023-08-07T17:20:51.272+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36 Edg/93.0.961.33', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:50.648Z', 'ecs': {'version': '8.0.0'}, 'request_length': 26, 'time_finish': '2023-08-07T17:20:51.272+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400050648]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'tq5Mz4kB6FVbQpdavwl7', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:20:58.724', 'server_name': 'cu-3.cn', 'geoip': {'timezone': 'Asia/Shanghai', 'ip': '182.147.60.2', 'latitude': 34.7732, 'country_name': 'China', 'country_code2': 'CN', 'country_code3': 'CN', 'continent_code': 'AS', 'location': {'lon': 113.722, 'lat': 34.7732}, 'longitude': 113.722}, 'time_start': '2023-08-07T17:20:58.650+08:00', 'level': 'error', 'access_status': 502, 'request_method': 'POST', 'remote_address': '182.147.60.2', 'message': "2023-08-07 17:20:58.724 | ERROR    | __main__:debug_log:46 - {'access_status': 502, 'request_method': 'POST', 'request_uri': '/login/', 'request_length': 70, 'remote_address': '182.147.60.2', 'server_name': 'cu-3.cn', 'time_start': '2023-08-07T17:20:58.650+08:00', 'time_finish': '2023-08-07T17:20:59.117+08:00', 'http_user_agent': 'Safari 11\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5'}", 'request_uri': '/login/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Safari 11\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:20:59.666Z', 'ecs': {'version': '8.0.0'}, 'request_length': 70, 'time_finish': '2023-08-07T17:20:59.117+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400059666]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': '7a5Mz4kB6FVbQpdawwlX', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:00.032', 'server_name': 'cm-2.cn', 'geoip': {'timezone': 'Asia/Seoul', 'latitude': 37.1579, 'ip': '182.219.137.40', 'continent_code': 'AS', 'city_name': 'Osan', 'country_code2': 'KR', 'country_name': 'South Korea', 'country_code3': 'KR', 'location': {'lon': 127.0662, 'lat': 37.1579}, 'region_name': 'Gyeonggi-do', 'postal_code': '181', 'longitude': 127.0662, 'region_code': '41'}, 'time_start': '2023-08-07T17:20:59.584+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '182.219.137.40', 'message': "2023-08-07 17:21:00.032 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/public/', 'request_length': 30, 'remote_address': '182.219.137.40', 'server_name': 'cm-2.cn', 'time_start': '2023-08-07T17:20:59.584+08:00', 'time_finish': '2023-08-07T17:21:00.192+08:00', 'http_user_agent': 'Firefox 7\\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Firefox 7\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:00.668Z', 'ecs': {'version': '8.0.0'}, 'request_length': 30, 'time_finish': '2023-08-07T17:21:00.192+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400060668]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': '965Mz4kB6FVbQpdaxwk_', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:00.933', 'server_name': 'cu-4.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '153.102.152.171', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:21:00.734+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '153.102.152.171', 'message': "2023-08-07 17:21:00.933 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/public/', 'request_length': 84, 'remote_address': '153.102.152.171', 'server_name': 'cu-4.cn', 'time_start': '2023-08-07T17:21:00.734+08:00', 'time_finish': '2023-08-07T17:21:01.392+08:00', 'http_user_agent': 'Firefox 7\\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Firefox 7\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:01.670Z', 'ecs': {'version': '8.0.0'}, 'request_length': 84, 'time_finish': '2023-08-07T17:21:01.392+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400061670]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': '-65Mz4kB6FVbQpdaywkk', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:01.896', 'server_name': 'cm-5.cn', 'geoip': {'timezone': 'Asia/Seoul', 'ip': '115.18.133.96', 'latitude': 37.5112, 'country_name': 'South Korea', 'country_code2': 'KR', 'country_code3': 'KR', 'continent_code': 'AS', 'location': {'lon': 126.9741, 'lat': 37.5112}, 'longitude': 126.9741}, 'time_start': '2023-08-07T17:21:01.735+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '115.18.133.96', 'message': "2023-08-07 17:21:01.896 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/login/', 'request_length': 74, 'remote_address': '115.18.133.96', 'server_name': 'cm-5.cn', 'time_start': '2023-08-07T17:21:01.735+08:00', 'time_finish': '2023-08-07T17:21:01.948+08:00', 'http_user_agent': 'Firefox 7\\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'}", 'request_uri': '/login/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Firefox 7\tMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:02.670Z', 'ecs': {'version': '8.0.0'}, 'request_length': 74, 'time_finish': '2023-08-07T17:21:01.948+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400062670]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Aa5Mz4kB6FVbQpda0gr7', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:03.941', 'server_name': 'cu-8.cn', 'geoip': {'timezone': 'Asia/Shanghai', 'ip': '220.114.225.226', 'latitude': 34.7732, 'country_name': 'China', 'country_code2': 'CN', 'country_code3': 'CN', 'continent_code': 'AS', 'location': {'lon': 113.722, 'lat': 34.7732}, 'longitude': 113.722}, 'time_start': '2023-08-07T17:21:03.054+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '220.114.225.226', 'message': "2023-08-07 17:21:03.941 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/account/', 'request_length': 76, 'remote_address': '220.114.225.226', 'server_name': 'cu-8.cn', 'time_start': '2023-08-07T17:21:03.054+08:00', 'time_finish': '2023-08-07T17:21:04.733+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:04.672Z', 'ecs': {'version': '8.0.0'}, 'request_length': 76, 'time_finish': '2023-08-07T17:21:04.733+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400064672]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Jq5Mz4kB6FVbQpda3grC', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:06.812', 'server_name': 'cu-8.cn', 'geoip': {'timezone': 'Asia/Tokyo', 'latitude': 36.7024, 'ip': '219.124.94.98', 'continent_code': 'AS', 'city_name': 'Toyama', 'country_code2': 'JP', 'country_name': 'Japan', 'country_code3': 'JP', 'location': {'lon': 137.2198, 'lat': 36.7024}, 'region_name': 'Toyama', 'postal_code': '930-0855', 'longitude': 137.2198, 'region_code': '16'}, 'time_start': '2023-08-07T17:21:06.656+08:00', 'level': 'error', 'access_status': 503, 'request_method': 'POST', 'remote_address': '219.124.94.98', 'message': "2023-08-07 17:21:06.812 | ERROR    | __main__:debug_log:46 - {'access_status': 503, 'request_method': 'POST', 'request_uri': '/account/', 'request_length': 95, 'remote_address': '219.124.94.98', 'server_name': 'cu-8.cn', 'time_start': '2023-08-07T17:21:06.656+08:00', 'time_finish': '2023-08-07T17:21:07.345+08:00', 'http_user_agent': 'Safari 11\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5'}", 'request_uri': '/account/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Safari 11\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:07.676Z', 'ecs': {'version': '8.0.0'}, 'request_length': 95, 'time_finish': '2023-08-07T17:21:07.345+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400067676]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'L65Mz4kB6FVbQpda7gpa', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:11.256', 'server_name': 'cu-5.cn', 'geoip': {'timezone': 'America/Toronto', 'ip': '198.168.208.16', 'latitude': 43.6319, 'country_name': 'Canada', 'country_code2': 'CA', 'country_code3': 'CA', 'continent_code': 'NA', 'location': {'lon': -79.3716, 'lat': 43.6319}, 'longitude': -79.3716}, 'time_start': '2023-08-07T17:21:10.726+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '198.168.208.16', 'message': "2023-08-07 17:21:11.256 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/login/', 'request_length': 96, 'remote_address': '198.168.208.16', 'server_name': 'cu-5.cn', 'time_start': '2023-08-07T17:21:10.726+08:00', 'time_finish': '2023-08-07T17:21:11.828+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36'}", 'request_uri': '/login/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:11.682Z', 'ecs': {'version': '8.0.0'}, 'request_length': 96, 'time_finish': '2023-08-07T17:21:11.828+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400071682]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'NK5Mz4kB6FVbQpda8go-', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:12.392', 'server_name': 'cu-2.cn', 'geoip': {'timezone': 'America/Chicago', 'ip': '129.176.58.105', 'latitude': 37.751, 'country_name': 'United States', 'country_code2': 'US', 'country_code3': 'US', 'continent_code': 'NA', 'location': {'lon': -97.822, 'lat': 37.751}, 'longitude': -97.822}, 'time_start': '2023-08-07T17:21:11.822+08:00', 'level': 'error', 'access_status': 503, 'request_method': 'GET', 'remote_address': '129.176.58.105', 'message': "2023-08-07 17:21:12.392 | ERROR    | __main__:debug_log:46 - {'access_status': 503, 'request_method': 'GET', 'request_uri': '/public/', 'request_length': 61, 'remote_address': '129.176.58.105', 'server_name': 'cu-2.cn', 'time_start': '2023-08-07T17:21:11.822+08:00', 'time_finish': '2023-08-07T17:21:13.259+08:00', 'http_user_agent': 'Mobile Safari\\tMozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Mobile Safari\tMozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:12.683Z', 'ecs': {'version': '8.0.0'}, 'request_length': 61, 'time_finish': '2023-08-07T17:21:13.259+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400072683]}, {'_index': '.ds-logs-myapp-default-2023.08.03-000001', '_id': 'Ya5Mz4kB6FVbQpda_Qr6', '_score': None, '_source': {'log_timestamp': '2023-08-07 17:21:14.724', 'server_name': 'cu-8.cn', 'geoip': {'timezone': 'Europe/Rome', 'latitude': 40.9986, 'ip': '151.26.7.206', 'continent_code': 'EU', 'city_name': 'Toritto', 'country_code2': 'IT', 'country_name': 'Italy', 'country_code3': 'IT', 'location': {'lon': 16.6867, 'lat': 40.9986}, 'region_name': 'Bari', 'postal_code': '70020', 'longitude': 16.6867, 'region_code': 'BA'}, 'time_start': '2023-08-07T17:21:14.339+08:00', 'level': 'error', 'access_status': 504, 'request_method': 'GET', 'remote_address': '151.26.7.206', 'message': "2023-08-07 17:21:14.724 | ERROR    | __main__:debug_log:46 - {'access_status': 504, 'request_method': 'GET', 'request_uri': '/public/', 'request_length': 31, 'remote_address': '151.26.7.206', 'server_name': 'cu-8.cn', 'time_start': '2023-08-07T17:21:14.339+08:00', 'time_finish': '2023-08-07T17:21:15.448+08:00', 'http_user_agent': 'Chrome 9\\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36'}", 'request_uri': '/public/', 'tags': ['beats_input_codec_plain_applied'], 'http_user_agent': 'Chrome 9\tMozilla/5.0 (Linux; Android 9; MX10 PRO) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36', 'input': {'type': 'log'}, '@timestamp': '2023-08-07T09:21:15.686Z', 'ecs': {'version': '8.0.0'}, 'request_length': 31, 'time_finish': '2023-08-07T17:21:15.448+08:00', 'data_stream': {'namespace': 'default', 'type': 'logs', 'dataset': 'myapp'}, 'host': {'name': '14f718adaba9'}, '@version': '1', 'class': '__main__:debug_log:46'}, 'sort': [1691400075686]}]}}

插入單條(索引)

往ES中插入數據時，如果索引不存在，會自動幫我們創建索引，使用動態mapping。

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def insert_data(self, index: str, data: dict) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param data: 數據內容
        :return:
        """
        logger.info(self.client.index(index=index, document=data))


if __name__ == '__main__':
    es = ES()
    data = {"username": "alex", "age": 30}
    es.insert_data('user-info', data)

結果

2023-08-07 20:09:23.146 | INFO     | __main__:insert_data:20 - {'_index': 'user-info', '_id': 'cNnmz4kBX4TmJtcG5aps', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 0, '_primary_term': 1}

discover查看

圖片

插入數據(數據流)

插入數據后默認會新建一個索引，如果需要插入數據到數據流中，可以通過以下兩種方式，但前提插入內容中必須有@timestamp字段，內容為標準UTC時間格式，示例數據如下：

{'username': 'alex', 'age': 30, '@timestamp': '2023-08-08T00:29:12.045Z'}

按數據流標準名稱格式插入數據

數據流標準名稱格式為data_stream_type(數據流類型，目前僅支持這幾種logs、metrics、synthetics、traces)-data_stream_dataset(數據流數據集，自定義字符串)-data_stream_namespace(數據流命名空間。默認為default)。例如插入的索引名稱為logs-userinfo-test，示例代碼如下：

from datetime import datetime

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def insert_data(self, index: str, data: dict) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param data: 數據內容
        :return:
        """
        logger.info(self.client.index(index=index, document=data))


if __name__ == '__main__':
    es = ES()
    data = {"username": "alex", "age": 30, "@timestamp": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"}
    logger.info(data)
    es.insert_data('logs-userinfo-test', data)

查看數據流信息

圖片

配置數據流模板后插入

如果有自定義命名需求的話，可以提前創建數據流模板，然后再插入數據到自定義的數據流名稱下。

# 創建數據流模板
PUT _index_template/user_info
{
  "index_patterns": ["user_info*"],
  "data_stream": { },
  "template": {}
}

插入數據到user_info中

from datetime import datetime

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def insert_data(self, index: str, data: dict) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param data: 數據內容
        :return:
        """
        logger.info(self.client.index(index=index, document=data))


if __name__ == '__main__':
    es = ES()
    data = {"username": "alex", "age": 30, "@timestamp": datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"}
    logger.info(data)
    es.insert_data('user_info', data)

效果如下

圖片

批量插入操作與上面類似。

更新

更新數據時，只需要指定索引名稱和文檔id以及文檔內容即可。

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def update(self, index: str, id: str, data: dict) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param index: 文檔ID
        :param data: 數據內容
        :return:
        """
        logger.info(self.client.update(index=index, id=id, doc=data))


if __name__ == '__main__':
    es = ES()
    data = {"username": "tom", "age": 35}
    es.update('user-info', 'cNnmz4kBX4TmJtcG5aps', data)

執行結果

2023-08-07 20:10:46.965 | INFO     | __main__:update:21 - {'_index': 'user-info', '_id': 'cNnmz4kBX4TmJtcG5aps', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 4, '_primary_term': 1}

discover查看

圖片

刪除

刪除數據時，只需要指定索引名稱和文檔id即可。

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def delete(self, index: str, id: str) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param index: 文檔ID
        :return:
        """
        logger.info(self.client.delete(index=index, id=id))


if __name__ == '__main__':
    es = ES()
    es.delete('user-info', 'cNnmz4kBX4TmJtcG5aps')

結果

2023-08-07 20:11:18.312 | INFO     | __main__:delete:20 - {'_index': 'user-info', '_id': 'cNnmz4kBX4TmJtcG5aps', '_version': 3, 'result': 'deleted', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 5, '_primary_term': 1}

discover

圖片

客戶端進階操作

針對ES的其他常用操作，官方為我們提供了幫助程序函數的集合，這些函數抽象了一些原始 API 的細節，需要注意的是，當我們需要傳入DSL查詢語句時，helpers模塊傳的還是原來的body參數。

滾動查詢全部

如果數據滿足條件的數據較多，想獲取全部數據，可采用如下方法：

size值調整超大：不夠靈活，數據量較大時容易查詢超時。
page+size參數查詢：查詢第n頁數據實際上es是把前n頁的數據全部找出來，再去除前n-1頁最后得到需要的數據返回，查最后一頁就相當于全掃描，效率低下。
滾動查詢：也叫游標查詢，非常適合大批量數據查詢。

接下來我們helpers來實現滾動查詢，代碼如下：

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def query_scan(self, index: str, query: dict) -> None:
        """
        獲取DSL滾動查詢結果的內容
        :param index:查詢索引
        :param query:查詢語句
        :return:
        """
        result = helpers.scan(self.client, query=query, index=index, scroll='2m', size=1000)
        data = []
        for i in result:
            data.append(i)
        logger.info(data)
        logger.info(len(data))


if __name__ == '__main__':
    es = ES()
    query = {"query": {"bool": {"filter": [{"terms": {"access_status": ["502", "503", "504"]}}, {
        "range": {"@timestamp": {"gte": "2023-08-07T17:20:00.000+08:00", "lt": "2023-08-07T17:30:00.000+08:00"}}}]}}}
    sort = [{"@timestamp": {"order": "asc"}}]
    es.query_scan('logs-myapp-default', query)

結果

2023-08-08 15:37:17.207 | INFO     | __main__:query_scan:24 - [查詢結果列表]
2023-08-08 15:37:17.215 | INFO     | __main__:query_scan:25 - 173

批量插入

我們經常有往ES集群寫入數據的需求，使用elasticsearch包只能單條逐個寫入，性能非常低下。此時可以借助helpers包實現數據批量插入。

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def insert_bulk(self, index: str, data: list) -> None:
        """
        ES中插入單條數據
        :param index: 索引
        :param data: 數據內容
        :return:
        """
        logger.info(helpers.bulk(self.client, data))


if __name__ == '__main__':
    es = ES()
    data = [{'_index': 'user-info', "username": "alex1", "age": 11},
            {'_index': 'user-info', "username": "alex2", "age": 12},
            {'_index': 'user-info', "username": "alex3", "age": 13}]
    es.insert_bulk('user-info', data)

結果

2023-08-07 19:45:05.499 | INFO     | __main__:insert_bulk:20 - (3, [])

discover查看數據

圖片

reindex

如果我們想查詢指定條件的數據，將結果寫入新的index中，使用elasticsearch包的話，需要調用查詢和寫入兩步操作，而借助helpers包，我們只需要調用reindex即可。

例如我們通過reindex操作把logs-myapp-default中的access_status字段值為502的日志復制到名為error-log的index中，DSL語法如下：

POST _reindex
{
  "source": {
    "index": "logs-myapp-default",
    "query": {
      "terms": {
        "access_status": [
          "502",
          "503",
          "504"
        ]
      }
    }
  },
  "dest": {
    "index": "error-log"
  }
}

通過python客戶端代碼實現如下：

from elasticsearch import Elasticsearch, helpers
from loguru import logger


class ES:
    def __init__(self):
        self.client = Elasticsearch(
            hosts='https://localhost:9200',
            ca_certs="certs/http_ca.crt",
            basic_auth=('elastic', 'ANrjs2w+upw0O1a1Yuka')
        )

    def reindex(self, source_index: str, target_index: str, query: dict) -> None:
        """
        ES reindex指定條件的數據到新的index中
        :param source_index: 原數據索引
        :param target_index: 目的數據索引
        :param query: 查詢條件
        :return:
        """
        logger.info(helpers.reindex(self.client, source_index=source_index, target_index=target_index, query=query))


if __name__ == '__main__':
    es = ES()
    query = {"query": {"bool": {"filter": [{"terms": {"access_status": ["502", "503", "504"]}}, {
        "range": {"@timestamp": {"gte": "2023-08-07T17:20:00.000+08:00", "lt": "2023-08-07T17:30:00.000+08:00"}}}]}}}
    es.reindex('logs-myapp-default', 'error-log', query)

執行結果

2023-08-08 15:42:05.370 | INFO     | __main__:reindex:31 - (173, 0)

kibana查看

圖片

集群操作

使用場景

在日常的運維中，我們除了對索引中的數據進行頻繁的操作外，也會對集群以及索引進行操作。此時Python客戶端就不能滿足我們的需求，我們可以通過httpx庫，調用es的api接口完成集群層面的操作。

例如我們可以開發定時任務或者故障自愈程序，當發現ES集群處于yellow或red狀態時，掃描出異常的索引分片，根據故障原因自動調整索引配置，從而解決故障。

圖片

使用示例

以下演示通過程序自動掃描處于yellow狀態的索引，并自動調整副本數完成故障處理。由于是演示效果，所有故障類型和故障解決方案很單一，實際生產情況中可能會遇到各種故障，我們可以更具不同的故障類型制定不同的解決方案。集群常見異常以及處理步驟參考文檔：https://www.cuiliangblog.cn/detail/section/82848020

import ssl
from json import JSONDecodeError

import httpx
from loguru import logger


class ES:
    def __init__(self):
        self.base_url = "https://localhost:9200"
        self.header = {'Content-type': 'application/json', 'kbn-xsrf': 'reporting'}
        self.auth = ('elastic', 'ANrjs2w+upw0O1a1Yuka')
        self.context = ssl.create_default_context()
        self.context.load_verify_locations(cafile='certs/http_ca.crt')

    @staticmethod
    def __format_result(res):
        if res.status_code == 200:
            try:
                logger.info(res.json())
                return res.json()
            except JSONDecodeError:
                logger.info(res.text)
                return res.text
        else:
            logger.error(res.raise_for_status())
            return None

    def get(self, url):
        res = httpx.get(self.base_url + url, headers=self.header, auth=self.auth, verify=self.context)
        return self.__format_result(res)

    def put(self, url, data):
        res = httpx.put(self.base_url + url, data=data, headers=self.header, auth=self.auth, verify=self.context)
        return self.__format_result(res)


if __name__ == '__main__':
    es = ES()
    # 獲取集群狀態
    es_status_result = es.get('/_cluster/health')
    es_status = es_status_result['status']
    logger.info("ES集群當前狀態為%s" % es_status)

    if es_status != 'green':
        # 檢查哪些索引異常
        error_indices_result = es.get('/_cat/indices?health=' + es_status)
        error_indices = []
        for i in error_indices_result.split("\n")[:-1]:
            error_indices.append(i.split(" ")[2])
        logger.info("ES集群異常索引分片名稱為%s" % error_indices)
        for indices in error_indices:
            # 更新索引副本數設置
            data = '{"index": {"number_of_replicas": 0}}'
            es.put('/' + indices + '/_settings', data)
            logger.info("已將%s索引副本數設為0" % indices)

執行結果

2023-08-08 17:37:35.262 | INFO     | __main__:__format_result:20 - {'cluster_name': 'docker-cluster', 'status': 'yellow', 'timed_out': False, 'number_of_nodes': 1, 'number_of_data_nodes': 1, 'active_primary_shards': 66, 'active_shards': 66, 'relocating_shards': 0, 'initializing_shards': 0, 'unassigned_shards': 2, 'delayed_unassigned_shards': 0, 'number_of_pending_tasks': 0, 'number_of_in_flight_fetch': 0, 'task_max_waiting_in_queue_millis': 0, 'active_shards_percent_as_number': 97.05882352941177}
2023-08-08 17:37:35.262 | INFO     | __main__:<module>:43 - ES集群當前狀態為yellow
2023-08-08 17:37:36.929 | INFO     | __main__:__format_result:23 - yellow open error-log  FRxfDibrSpGRx4RB-XuuUw 1 1 173 0 173.9kb 173.9kb
yellow open test_index ZucbZ71pT6CGpZ9H96OvYw 1 1   0 0    247b    247b

2023-08-08 17:37:36.929 | INFO     | __main__:<module>:51 - ES集群異常索引分片名稱為['error-log', 'test_index']
2023-08-08 17:37:37.147 | INFO     | __main__:__format_result:20 - {'acknowledged': True}
2023-08-08 17:37:37.147 | INFO     | __main__:<module>:56 - 已將error-log索引副本數設為0
2023-08-08 17:37:37.285 | INFO     | __main__:__format_result:20 - {'acknowledged': True}
2023-08-08 17:37:37.286 | INFO     | __main__:<module>:56 - 已將test_index索引副本數設為0

接下來我們查看Kibana驗證，已經所有異常索引副本數改為1，集群恢復正常。

參考文檔

python客戶端連接ES配置：

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html

Python操作ES常用示例：

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/examples.html

helpers模塊使用示例：

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/client-helpers.html

原文鏈接：https://www.cuiliangblog.cn/

責任編輯：武曉燕來源：崔亮的博客

成人免费xxxxx在线视频软件_久久精品久久久_亚洲国产精品久久久_天天色天天色_亚洲人成一区_欧美一级欧美三级在线观看

ELK Stack生產實踐——Python操作Elasticsearch

安裝與連接

安裝

使用CA證書+Basic Auth連接

使用證書指紋+Basic Auth連接

使用CA證書+token連接

使用CA證書+key連接

客戶端基礎操作

查詢結果統計

DSL查詢

插入單條(索引)

插入數據(數據流)

按數據流標準名稱格式插入數據

配置數據流模板后插入

更新

刪除

客戶端進階操作

滾動查詢全部

批量插入

reindex

集群操作

使用場景

使用示例

參考文檔