If I send the CONT signal to fluentbit I see that fluentbit still has them. [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_main-4522cea91646c207c4aa9ad008d19d9620bc8c6a81ae6135922fb2d99ee834c7.log, inode 34598706 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [retry] re-using retry for task_id=2 attempts=3 Fri, Mar 25 2022 3:08:37 pm | [2022/03/25 07:08:37] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY What versions are you using? Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fluentbit stops sending data to output. [2022/03/24 04:19:30] [debug] [retry] re-using retry for task_id=2 attempts=2 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-89knq_argo_main-f011b1f724e7c495af7d5b545d658efd4bff6ae88489a16581f492d744142807.log, inode 35326801 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [input chunk] update output instances with new chunk size diff=1085 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"AeMmun8BI6SaBP9l_8rZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:54] [debug] [retry] re-using retry for task_id=0 attempts=3 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Create a new Azure Storage account. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY with the updated value.yaml file. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"1OMmun8BI6SaBP9liJWQ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"YeMnun8BI6SaBP9lLtm1","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available [2022/03/24 04:20:51] [error] [outputes.0] could not pack/validate JSON response Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [retry] new retry created for task_id=16 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input chunk] update output instances with new chunk size diff=1167 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_wait-6b82c7411c8433b5e5f14c56f4b810dc3e25a2e7cfb9e9b107b9b1d50658f5e2.log, inode 67891711 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/workflow-controller-bb7c78c7b-w2n5c_argo_workflow-controller-7f4797ff53352e50ff21cf9625ec02ffb226172a2a3ed9b0cee0cb1d071a2990.log, inode 34598688 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [input chunk] update output instances with new chunk size diff=1083 Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [out coro] cb_destroy coro_id=8 [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=35326801 with offset=0 appended as /var/log/containers/hello-world-89knq_argo_main-f011b1f724e7c495af7d5b545d658efd4bff6ae88489a16581f492d744142807.log Please . Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input chunk] update output instances with new chunk size diff=681 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY It diagnoses problems by analyzing hundreds of metrics collected by a lightweight agent and offers guidance for resolving them. Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [input chunk] update output instances with new chunk size diff=862 Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available 1 chunk_idle_period: 2m chunk_block_size: 2621440 chunk_encoding: snappy chunk_retain_period: 1m max_transfer_retries: 0 wal: enabled: true dir: /var/loki/wal limits_config: enforce_metric_name: false reject_old_samples . Improve this answer. counter. [2022/03/24 04:20:51] [ warn] [engine] failed to flush chunk '1-1648095560.205735907.flb', retry in 111 seconds: task_id=0, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available [2022/03/24 04:21:20] [debug] [input:tail:tail.0] inode=1772861 with offset=0 appended as /var/log/containers/hello-world-wpr5j_argo_wait-76bcd0771f3cc7b5f6b5f15f16ee01cc0c671fb047b93910271bc73e753e26ee.log The most common reason i've seen for "failed to flush chunk" is that the batch upload queue on the ES cluster is full. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input chunk] update output instances with new chunk size diff=862 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:22] [debug] [input chunk] update output instances with new chunk size diff=641 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY [2022/03/24 04:19:50] [debug] [upstream] KA connection #102 to 10.3.4.84:9200 is now available "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"--Mmun8BI6SaBP9l_8nZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [ warn] [engine] failed to flush chunk '1-1648192121.87279162.flb', retry in 6 seconds: task_id=15, input=tail.0 > output=es.0 (out_id=0), Hi, @lecaros still got error, already add setting "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IeMmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"I-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"0-Mnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"KOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Under 200 tps everything is working . Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] task_id=14 assigned to thread #0 Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [outputes.0] task_id=6 assigned to thread #0 Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [outputes.0] task_id=17 assigned to thread #0 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [out coro] cb_destroy coro_id=15 Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=3076476 events: IN_ATTRIB Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Name es Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [retry] new retry created for task_id=8 attempts=1 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69479190 removing file name /var/log/containers/hello-world-dsfcz_argo_main-13bb1b2c7e9d3e70003814aa3900bb9aef645cf5e3270e3ee4db0988240b9eff.log "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","id":"-Mmun8BI6SaBP9l_8nZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:49] [debug] [outputes.0] task_id=0 assigned to thread #1 [OUTPUT] [2022/03/24 04:19:20] [debug] [input chunk] tail.0 is paused, cannot append records Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [out coro] cb_destroy coro_id=16 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [upstream] KA connection #120 to 10.3.4.84:9200 has been assigned (recycled) Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:31 pm | [2022/03/25 07:08:31] [debug] [retry] new retry created for task_id=9 attempts=1 Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=655 next_retry=2019-01-27 19:00:14 -0500 error_class="ArgumentError" error="Data too big (189382 bytes), would create more than 128 chunks!" plugin_id="object:3fee25617fbc" Because of this cache memory increases and td-agent fails to send messages to graylog
Judy Holliday Net Worth At Death,
Dawson County Schools Board Minutes,
Raymond Moore Obituary,
Healthy Options At Kobe Steakhouse,
How Much Does Scott D Henry Make On Tiktok,
Articles F