1 Tomcat 概述

Tomcat隶属于Apache基金会，是开源的轻量级Web应用服务器，使用非常广泛。

1.0 Tomcat 容器与原理

1.0.1 Tomcat组件构成

注意，如图所示，阴影部分代表存在多个同样元素。

Tomcat 主要由 Server、Service、Engine、Host、Context 等部分组成，介绍如下。

Server：服务器 Tomcat 的顶级元素。

一个 Tomcat 中只有一个 Server

Service：Engine（引擎）的集合，包括线程池 Executor 和连接器 Connector 的定义。

一个 Server 可以包含多个 Service

一个 Service 可以有一个 Engine、及一组与之相连的多个Connector

Service 对外提供服务，Service 的生命周期由 Server 管理控制。

Engine（引擎、Servlet容器）：Engine 代表一个完整的 Servlet 引擎，它接收来自 Connector 的请求，并决定传给哪个 Host 来处理。

一个 Service 可以有一个 Engine

容器的功能是处理Connector接收进来的请求，并产生相应的响应。
Engine、Host和Context都是容器，但它们不是平行的关系，而是父子关系：Engine包含Host，Host包含Context。
一个Engine组件可以处理Service中的所有请求，一个Host组件可以处理发向一个特定虚拟主机的所有请求，一个Context组件可以处理一个特定Web应用的所有请求。

Container（容器接口）：Host、Context、Engine 和 Wraper 都继承自 Container 接口，它们都是容器。

一个 Service 可以有多个 Connector

即: 一个服务可有多个 Connector

Connector（连接器）：将 Service 和 Container 连接起来，注册到一个 Service，把来自客户端的请求转发到 Container。

<Connector>代表了外部客户端发送请求到特定Service的接口；同时，也是外部客户端从特定Service接收响应的接口。

Host：站点（虚拟机），通过配置 Host 可以添加站点。

Context：Web 应用程序，一个 Context 即对应一个 Web 应用程序。

Context 容器直接管理 Servlet 的运行，将 Servlet 包装成一个 StandardWrapper 类去运行。

Wrapper 负责管理一个 Servlet 的装载、初始化、执行以及资源回收，是最底层容器。

Listener：监听器。

Resource：配置 Tomcat 数据源。

JNDI：它是一个为 Java 应用程序提供命名服务的应用程序接口，为我们提供了查找和访问各种命名和目录服务的通用统一的接口。通过 JNDI 统一接口，我们可以访问各种不同类型的服务。

1.0.2 server.xml

server.xml是Tomcat中最重要的配置文件，server.xml的每一个元素都对应了Tomcat中的一个组件；
通过对xml文件中元素的配置，可以实现对Tomcat中各个组件的控制。
server.xml位于$TOMCAT_HOME/conf目录下
server.xml的整体结构大致如下：

<Server>

     <Service>

        <Connector />

        <Connector />

         <Engine>

             <Host>

                 <Context /><!-- 现在常常使用自动部署，不推荐配置Context元素，Context小节有详细说明 -->

            </Host>

         </Engine>

    </Service>

</Server>

该结构中只给出了Tomcat的核心组件，除了核心组件外，Tomcat还有一些其他组件。

Tomcat 容器结构配置 server.xml 代码清单如下所示。

<?xml version="1.0" encoding="UTF-8"?>

<Server port="8005" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />

  <!-- Security listener. Documentation at /docs/config/listeners.html

  <Listener className="org.apache.catalina.security.SecurityListener" />

  -->

  <!--APR library loader. Documentation at /docs/apr.html -->

  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

  <!-- Prevent memory leaks due to use of particular java/javax APIs-->

  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />

  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <!-- Global JNDI resources

       Documentation at /docs/jndi-resources-howto.html

  -->

  <GlobalNamingResources>

    <!-- Editable user database that can also be used by

         UserDatabaseRealm to authenticate users

    -->

    <Resource name="UserDatabase" auth="Container"

              type="org.apache.catalina.UserDatabase"

              description="User database that can be updated and saved"

              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"

              pathname="conf/tomcat-users.xml" />

  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share

       a single "Container" Note:  A "Service" is not itself a "Container",

       so you may not define subcomponents such as "Valves" at this level.

       Documentation at /docs/config/service.html

   -->

  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->

    <!--

    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"

        maxThreads="150" minSpareThreads="4"/>

    -->

    <!-- A "Connector" represents an endpoint by which requests are received

         and responses are returned. Documentation at :

         Java HTTP Connector: /docs/config/http.html

         Java AJP  Connector: /docs/config/ajp.html

         APR (HTTP/AJP) Connector: /docs/apr.html

         Define a non-SSL/TLS HTTP/1.1 Connector on port 8080

    -->

    <Connector port="8077" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    <!-- A "Connector" using the shared thread pool-->

    <!--

    <Connector executor="tomcatThreadPool"

               port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    -->

    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443

         This connector uses the NIO implementation. The default

         SSLImplementation will depend on the presence of the APR/native

         library and the useOpenSSL attribute of the

         AprLifecycleListener.

         Either JSSE or OpenSSL style configuration may be used regardless of

         the SSLImplementation selected. JSSE style configuration is used below.

    -->

    <!--

    <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"

               maxThreads="150" SSLEnabled="true">

        <SSLHostConfig>

            <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"

                         type="RSA" />

        </SSLHostConfig>

    </Connector>

    -->

    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2

         This connector uses the APR/native implementation which always uses

         OpenSSL for TLS.

         Either JSSE or OpenSSL style configuration may be used. OpenSSL style

         configuration is used below.

    -->

    <!--

    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"

               maxThreads="150" SSLEnabled="true" >

        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />

        <SSLHostConfig>

            <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"

                         certificateFile="conf/localhost-rsa-cert.pem"

                         certificateChainFile="conf/localhost-rsa-chain.pem"

                         type="RSA" />

        </SSLHostConfig>

    </Connector>

    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

    <Engine name="Catalina" defaultHost="localhost">

      <Realm className="org.apache.catalina.realm.LockOutRealm">

        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"

               resourceName="UserDatabase"/>

      </Realm>

      <Host name="localhost"  appBase="webapps"

            unpackWARs="true" autoDeploy="true">

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

               prefix="localhost_access_log" suffix=".txt"

               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>

    </Engine>

  </Service>

</Server>

1.0.3 如何确定请求由谁处理？

当请求被发送到Tomcat所在的主机时，如何确定最终哪个Web应用来处理该请求呢？

（1）根据协议和端口号选定Service和Engine

Service中的Connector组件可以接收特定端口的请求，因此，当Tomcat启动时，Service组件就会监听特定的端口。在第一部分的例子中，Catalina这个Service监听了8080端口（基于HTTP协议）和8009端口（基于AJP协议）。当请求进来时，Tomcat便可以根据协议和端口号选定处理请求的Service；Service一旦选定，Engine也就确定。

通过在Server中配置多个Service，可以实现通过不同的端口号来访问同一台机器上部署的不同应用。

（2）根据域名或IP地址选定Host

Service确定后，Tomcat在Service中寻找名称与域名/IP地址匹配的Host处理该请求。如果没有找到，则使用Engine中指定的defaultHost来处理该请求。在第一部分的例子中，由于只有一个Host（name属性为localhost），因此该Service/Engine的所有请求都交给该Host处理。

（3）根据URI选定Context/Web应用

这一点在Context一节有详细的说明：Tomcat根据应用的 path属性与URI的匹配程度来选择Web应用处理相应请求，这里不再赘述。

（4）举例

以请求http://localhost:8080/app1/index.html为例，首先通过协议和端口号（http和8080）选定Service；然后通过主机名（localhost）选定Host；然后通过uri（/app1/index.html）选定Web应用。

1.0.4 如何配置多个服务?

通过在Server中配置多个Service服务，可以实现通过不同的端口号来访问同一台机器上部署的不同Web应用。

在server.xml中配置多服务的方法非常简单，分为以下几步：

（1）复制<Service>元素，放在当前<Service>后面。
（2）修改端口号：根据需要监听的端口号修改<Connector>元素的port属性；必须确保该端口没有被其他进程占用，否则Tomcat启动时会报错，而无法通过该端口访问Web应用。

以Win7为例，可以用如下方法找出某个端口是否被其他进程占用：netstat -aon|findstr "8081"，发现8081端口被PID为2064的进程占用，tasklist | findstr "2064"发现该进程为FrameworkService.exe(这是McAfee杀毒软件的进程)。

（3）修改Service和Engine的name属性

（4）修改Host的appBase属性（如webapps2）

（5）Web应用仍然使用自动部署

（6）将要部署的Web应用(WAR包或应用目录)拷贝到新的appBase下。

以第一部分的server.xml为例，多个Service的配置如下：

<?xml version='1.0' encoding='utf-8'?>   <Server port="8005" shutdown="SHUTDOWN">

    <Listener className="org.apache.catalina.startup.VersionLoggerListener" />

    <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

    <Listener className="org.apache.catalina.core.JasperListener" />

    <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />

    <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

    <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />  

   <GlobalNamingResources>

     <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />

   </GlobalNamingResources>  

   <Service name="Catalina">

     <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />

     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

     <Engine name="Catalina" defaultHost="localhost">

       <Realm className="org.apache.catalina.realm.LockOutRealm">

         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"

                resourceName="UserDatabase"/>

       </Realm>  

       <Host name="localhost"  appBase="/opt/project/webapps" unpackWARs="true" autoDeploy="true">

         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" />

       </Host>

     </Engine>

   </Service>  

   <Service name="Catalina2">

     <Connector port="8084" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />

     <Connector port="8010" protocol="AJP/1.3" redirectPort="8443" />

     <Engine name="Catalina2" defaultHost="localhost">

       <Realm className="org.apache.catalina.realm.LockOutRealm">

         <Realm className="org.apache.catalina.realm.UserDatabaseRealm"

                resourceName="UserDatabase"/>

       </Realm>  

       <Host name="localhost"  appBase="/opt/project/webapps2" unpackWARs="true" autoDeploy="true">

         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" />

       </Host>

     </Engine>

   </Service>

</Server>

再将原webapps下的docs目录拷贝到webapps2中，则通过如下两个接口都可以访问docs应用：

http://localhost:8080/docs/

http://localhost:8084/docs/

1.0.5 其他组件 : Listener

除核心组件外，server.xml中还可以配置很多其他组件。下面只介绍第一部分例子中出现的组件，如果要了解更多内容，可以查看Tomcat官方文档。

Listener

<Listener className="org.apache.catalina.startup.VersionLoggerListener" />

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

<Listener className="org.apache.catalina.core.JasperListener" />

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />

<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

Listener(即监听器)定义的组件，可以在特定事件发生时执行特定的操作；被监听的事件通常是Tomcat的启动和停止。

监听器可以在Server、Engine、Host或Context中，本例中的监听器都是在Server中。实际上，本例中定义的6个监听器，都只能存在于Server组件中。监听器不允许内嵌其他组件。

监听器需要配置的最重要的属性是className，该属性规定了监听器的具体实现类，该类必须实现了org.apache.catalina.LifecycleListener接口。

下面依次介绍例子中配置的监听器：

VersionLoggerListener：当Tomcat启动时，该监听器记录Tomcat、Java和操作系统的信息。该监听器必须是配置的第一个监听器。

AprLifecycleListener：Tomcat启动时，检查APR库，如果存在则加载。APR，即Apache Portable Runtime，是Apache可移植运行库，可以实现高可扩展性、高性能，以及与本地服务器技术更好的集成。

JasperListener：在Web应用启动之前初始化Jasper，Jasper是JSP引擎，把JVM不认识的JSP文件解析成java文件，然后编译成class文件供JVM使用。

JreMemoryLeakPreventionListener：与类加载器导致的内存泄露有关。

GlobalResourcesLifecycleListener：通过该监听器，初始化< GlobalNamingResources>标签中定义的全局JNDI资源；如果没有该监听器，任何全局资源都不能使用。< GlobalNamingResources>将在后文介绍。

ThreadLocalLeakPreventionListener：当Web应用因thread-local导致的内存泄露而要停止时，该监听器会触发线程池中线程的更新。当线程执行完任务被收回线程池时，活跃线程会一个一个的更新。只有当Web应用(即Context元素)的renewThreadsWhenStoppingContext属性设置为true时，该监听器才有效。

1.0.6 其他组件 : GlobalNamingResources与Realm(域)

第一部分的例子中，Engine组件下定义了Realm组件：

<Realm className="org.apache.catalina.realm.LockOutRealm">

    <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>

</Realm>

Realm，可以把它理解成“域”；

Realm提供了一种用户密码与web应用的映射关系，从而达到角色安全管理的作用。

在本例中，Realm的配置使用name为UserDatabase的资源实现。而该资源在Server元素中使用GlobalNamingResources配置：

<GlobalNamingResources>

  <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />

</GlobalNamingResources>

GlobalNamingResources元素定义了全局资源，通过配置可以看出，该配置是通过读取$TOMCAT_HOME/conf/tomcat-users.xml实现的。

1.0.7 其他组件 : Valve(阀门)

在第一部分的例子中，Host元素内定义了Valve组件：

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" />

单词Valve的意思是“阀门”，在Tomcat中代表了请求处理流水线上的一个组件；

Valve可以与Tomcat的容器(Engine、Host或Context)关联。

不同的Valve有不同的特性，下面介绍一下本例中出现的AccessLogValve。

AccessLogValve的作用是通过日志记录其所在的容器中处理的所有请求，在本例中，Valve放在Host下，便可以记录该Host处理的所有请求。AccessLogValve记录的日志就是访问日志，每天的请求会写到一个日志文件里。AccessLogValve可以与Engine、Host或Context关联；在本例中，只有一个Engine，Engine下只有一个Host，Host下只有一个Context，因此AccessLogValve放在三个容器下的作用其实是类似的。

本例的AccessLogValve属性的配置，使用的是默认的配置；下面介绍AccessLogValve中各个属性的作用：

（1）className：规定了Valve的类型，是最重要的属性；本例中，通过该属性规定了这是一个AccessLogValve。

（2）directory：指定日志存储的位置，本例中，日志存储在$TOMCAT_HOME/logs目录下。

（3）prefix：指定了日志文件的前缀。

（4）suffix：指定了日志文件的后缀。通过directory、prefix和suffix的配置，在$TOMCAT_HOME/logs目录下，可以看到如下所示的日志文件。

（5）pattern：指定记录日志的格式，本例中各项的含义如下：

%h：远程主机名或IP地址；如果有nginx等反向代理服务器进行请求分发，该主机名/IP地址代表的是nginx，否则代表的是客户端。后面远程的含义与之类似，不再解释。

%l：远程逻辑用户名，一律是”-”，可以忽略。

%u：授权的远程用户名，如果没有，则是”-”。

%t：访问的时间。

%r：请求的第一行，即请求方法(get/post等)、uri、及协议。

%s：响应状态，200,404等等。

%b：响应的数据量，不包括请求头，如果为0，则是””-。

例如，下面是访问日志中的一条记录

pattern的配置中，除了上述各项，还有一个非常常用的选项是%D，含义是请求处理的时间(单位是毫秒)，对于统计分析请求的处理速度帮助很大。

开发人员可以充分利用访问日志，来分析问题、优化应用。例如，分析访问日志中各个接口被访问的比例，不仅可以为需求和运营人员提供数据支持，还可以使自己的优化有的放矢；分析访问日志中各个请求的响应状态码，可以知道服务器请求的成功率，并找出有问题的请求；分析访问日志中各个请求的响应时间，可以找出慢请求，并根据需要进行响应时间的优化。

1.0.8 Tomcat 文件目录结构

以下是 Tomcat 8 主要目录结构

1.1 引言: Tomcat - HTTP Connector

在使用tomcat时，经常会遇到连接数、线程数之类的配置问题，要真正理解这些概念，必须先了解Tomcat的连接器（Connector）。

Tomcat配置文件server.xml中:

Connector的主要功能:

Step1 接收连接请求，创建Request和Response对象用于和请求端交换数据；
Step2 分配线程让Engine（也就是Servlet容器）来处理这个请求，并把产生的Request和Response对象传给Engine。
Step3 当Engine处理完请求后，也会通过Connector将响应返回给客户端。

可以说，Servlet容器处理请求，是需要Connector进行调度和控制的，Connector是Tomcat处理请求的主干。

因此,Connector的配置和使用对Tomcat的性能有着重要的影响。

这篇文章将从Connector入手，讨论一些与Connector有关的重要问题，包括NIO/BIO模式、线程池、连接数等。

根据协议的不同，Connector可以分为:

HTTP Connector
AJP Connector等

本文只讨论HTTP Connector

1.2 Nio / Bio / APR

1.2.1 Connector 的 http protocol

Connector在处理HTTP请求时，会使用不同的protocol。不同的Tomcat版本支持的protocol不同，其中最典型的protocol包括:

BIO : Blocking IO，阻塞的IO
NIO : Non-blocking IO，非阻塞的IO
APR : Apache Portable Runtime，是Apache可移植运行库，利用本地库可以实现高可扩展性、高性能；是在Tomcat上运行高并发应用的首选模式，但是需要安装apr、apr-utils、tomcat-native等包。

Tomcat7中支持这3种；Tomcat8增加了对NIO2的支持；而到了Tomcat8.5和Tomcat9.0，则去掉了对BIO的支持。

1.2.2 如何指定protocol？

Connector使用哪种protocol，可以通过<connector>元素中的protocol属性进行指定，也可以使用默认值。指定的protocol取值及对应的协议如下：

HTTP/1.1：默认值，使用的协议与Tomcat版本有关

org.apache.coyote.http11.Http11Protocol：BIO
org.apache.coyote.http11.Http11NioProtocol：NIO
org.apache.coyote.http11.Http11Nio2Protocol：NIO2
org.apache.coyote.http11.Http11AprProtocol：APR

如果没有指定protocol，则使用默认值HTTP/1.1。其含义如下：

在Tomcat7中，自动选取使用BIO或APR（如果找到APR需要的本地库，则使用APR，否则使用BIO）
在Tomcat8中，自动选取使用NIO或APR（如果找到APR需要的本地库，则使用APR，否则使用NIO）

1.2.3 BIO/NIO有何不同?

无论是BIO，还是NIO，Connector处理请求的大致流程是一样的：

在accept队列中接收连接（当客户端向服务器发送请求时，如果客户端与OS完成三次握手建立了连接，则OS将该连接放入accept队列）；
在连接中获取请求的数据，生成request；
调用servlet容器(Enginee)处理request；
返回response。

为了便于后面的说明，首先明确一下连接与请求的关系：

连接是TCP层面的（传输层），对应socket；
请求是HTTP层面的（应用层），必须依赖于TCP的连接(Socket)实现；一个TCP连接中可能传输多个HTTP请求。

在BIO实现的Connector中，处理请求的主要实体是JIoEndpoint对象。JIoEndpoint维护了Acceptor和Worker：

Step1 Acceptor接收socket，
Step2 然后，从Worker线程池中找出空闲的线程处理socket。

如果worker线程池没有空闲线程，则：Acceptor将阻塞。其中，Worker是Tomcat自带的线程池，如果通过<Executor>配置了其他线程池，原理与Worker类似。

在NIO实现的Connector中，处理请求的主要实体是NIoEndpoint对象。NIoEndpoint中除了包含Acceptor和Worker外，还是用了Poller，处理流程如下图所示:

Acceptor接收socket后，不是直接使用Worker中的线程处理请求，而是先将请求发送给了Poller（Poller是实现NIO的关键）

Acceptor向Poller发送请求通过队列实现，使用了典型的生产者-消费者模式
在Poller中，维护了一个Selector对象

当Poller从队列中取出socket后，注册到该Selector中；
然后，通过遍历Selector，找出其中可读的socket，并使用Worker中的线程处理相应请求。

与BIO类似，Worker也可以被自定义的线程池代替。

通过上述过程可以看出，在NIoEndpoint处理请求的过程中，无论是Acceptor接收socket，还是线程处理请求，使用的仍然是阻塞方式；

但在读取socket，并交给Worker中的线程的这个过程中，使用非阻塞的NIO实现。

这是NIO模式与BIO模式的最主要区别（其他区别对性能影响较小，暂时略去不提）。

而这个区别，在并发量较大的情形下可以带来Tomcat效率的显著提升：

目前大多数HTTP请求使用的是长连接（HTTP/1.1，默认keep-alive为true）。
而长连接意味着，一个TCP的socket在当前请求结束后，如果没有新的请求到来，socket不会立马释放，而是等timeout后再释放。

如果使用BIO，读取socket并交给Worker中的线程这个过程是阻塞的。

也就意味着，在socket等待下一个请求或等待释放的过程中，处理这个socket的工作线程会一直被占用，无法释放
因此，Tomcat可以同时处理的socket数目不能超过最大线程数，性能受到了极大限制。

而使用NIO，读取socket并交给Worker中的线程这个过程是非阻塞的。
当socket在等待下一个请求或等待释放时，并不会占用Worker线程(工作线程)。
因此，Tomcat可以同时处理的socket数目远大于最大线程数，并发性能大大提高。

1.3 acceptCount、maxConnections、maxThreads

再回顾一下Tomcat处理请求的过程：

在accept队列中接收连接（当客户端向服务器发送请求时，如果客户端与OS完成三次握手建立了连接，则：OS将该连接放入accept队列）
在连接(Connection)中获取请求(Request)的数据，生成request；
调用servlet容器(Enginee)处理请求；
返回response

相对应的，Connector中的几个参数功能如下：

1.3.1 acceptCount

参数定义：accept队列的长度；

当accept队列中连接的个数达到acceptCount时，队列满，进来的请求一律被拒绝。

参数默认值: 100

1.3.2 maxConnections

参数定义: Tomcat在任意时刻接收和处理的最大连接数。

允许最大连接数。当达到maxConnections临界值时,系统可能会基于accept-count继续接受连接

当Tomcat接收的连接数达到maxConnections时，Acceptor线程不会读取accept队列中的连接(Connection)；

这时accept队列中的线程会一直阻塞着，直到Tomcat接收的连接数小于maxConnections。如果设置为-1，则:连接数不受限制。

参数默认值：

默认值与连接器使用的协议有关：

NIO的默认值: 10000
APR/native的默认值: 8192
BIO的默认值: maxThreads
如果配置了Executor，则默认值 : Executor的maxThreads

在windows下，APR/native的maxConnections值会自动调整为设置值以下最大的1024的整数倍；

如: 设置为2000，则: 最大值实际是1024

1.3.3 maxThreads

参数定义：请求处理线程的最大数量。

maxThreads规定的是最大的线程数目，并不是实际running的CPU数量
实际上，maxThreads的大小比CPU核心数量要大得多。

这是因为，处理请求的线程真正用于计算的时间可能很少，大多数时间可能在阻塞，如等待数据库返回数据、等待硬盘读写数据等。

因此，在某一时刻，只有少数的线程真正的在使用物理CPU，大多数线程都在等待。因此，线程数远大于物理CPU核心数才是合理的

参数默认值：200

Tomcat7和8都是200

如果该Connector绑定了Executor，这个值会被忽略。因为该Connector将使用绑定的Executor，而不是内置的线程池来执行任务。

换句话说，Tomcat通过使用比CPU核心数量多得多的线程数，可以使CPU忙碌起来，大大提高CPU的利用率。

1.3.4 minSpareThreads

参数定义 : 线程池维持最小线程

参数默认值 : 10

1.3.5 参数设置小结

maxThreads的设置既与应用的特点有关，也与服务器的CPU核心数量有关。

通过前面介绍可以知道，maxThreads数量应该远大于CPU核心数量；而且CPU核心数越大，maxThreads应该越大；应用中CPU越不密集（IO越密集），maxThreads应该越大，以便能够充分利用CPU。当然，maxThreads的值并不是越大越好，如果maxThreads过大，那么CPU会花费大量的时间用于线程的切换，整体效率会降低。

maxConnections的设置与Tomcat的运行模式有关。

如果tomcat使用的是BIO，那么maxConnections的值应该与maxThreads一致；如果tomcat使用的是NIO，那么类似于Tomcat的默认值，maxConnections值应该远大于maxThreads。

通过前面的介绍可以知道，虽然tomcat同时可以处理的连接数目是maxConnections，但服务器中可以同时接收的连接数为maxConnections+acceptCount 。

acceptCount的设置，与应用在连接过高情况下，希望做出什么反应有关系。

如果设置过大，后面进入的请求等待时间会很长；如果设置过小，后面进入的请求立马返回connection refused。

1.4 线程池 Executor

1.4.1 Executor 线程池

Executor元素代表Tomcat中的线程池，可以由其他组件共享使用；
要使用该线程池，组件需要通过executor属性指定该线程池。
Executor是Service元素的内嵌元素。

1.4.2 Executor 与 Connector

一般来说，使用线程池的是Connector组件；为了使Connector能使用线程池，Executor元素应该放在Connector前面。

Executor与Connector的配置举例如下：

<Executor name="tomcatThreadPool" namePrefix ="catalina-exec-" maxThreads="150" minSpareThreads="4" />

<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" acceptCount="1000" />

1.4.3 Executor的主要参数

Executor的主要属性包括：

name ：该线程池的标记

maxThreads ：线程池中最大活跃线程数，默认值200（Tomcat7和8都是）

minSpareThreads ：线程池中保持的最小线程数，最小值是25

maxIdleTime：线程空闲的最大时间，当空闲超过该值时关闭线程（除非线程数小于minSpareThreads），单位是ms，默认值60000（1分钟）

daemon ：是否后台线程，默认值true

threadPriority ：线程优先级，默认值5

namePrefix：线程名字的前缀

线程池中线程名字为：namePrefix+线程编号

2 监测Tomcat当前运行状态

2.1 简述

上面介绍了Tomcat的连接数、线程数的概念以及如何设置；下面说明如何查看服务器中的连接数和线程数。

查看服务器的状态，大致分为2种方案：

1 使用现成的工具

JDK jconsole
JDK jvisualvm / or VisualVM
Tomcat 自带的 manager
收费工具New Relic
...

2 直接使用Linux的命令查看

2.2 JDK

2.2.1 jconsole (线程信息等)

下图是jconsole查看线程信息的界面：

2.2.2 jstack

jstack -l <pid>

参见本文档 Y.1.1 `jstack -l `

2.3 VisualVM (线程信息等)

2.4 Linux Shell方式

下面说一下如何通过Linux命令行，查看服务器中的连接数和线程数。

2.4.1 连接数

假设Tomcat接收http请求的端口是8083，则可以使用如下语句查看连接情况：

netstat –nat | grep 8083

结果如下所示：

可以看出，有一个连接处于listen状态，监听请求；除此之外，还有4个已经建立的连接（ESTABLISHED）和2个等待关闭的连接（CLOSE_WAIT）。

2.4.2 线程

ps命令可以查看进程状态，如执行如下命令

ps -e | grep java

结果如下图：

可以看到，只打印了一个进程的信息：

27989是进程id，java是指执行的java命令。
这是因为启动一个tomcat，内部所有的工作都在这一个进程里完成，包括：

主线程
垃圾回收线程
Acceptor线程
请求处理线程
...

通过如下命令，可以看到该进程内有多少个线程；其中，nlwp含义是number of light-weight process。

ps -o nlwp 27989

可以看到，该进程内部有73个线程；但是73并没有排除处于idle状态的线程。要想获得真正在running的线程数量，可以通过以下语句完成：

ps -eLo pid ,stat | grep 27989 | grep running | wc -l

其中:

ps -eLo pid ,stat可以找出所有线程，并打印其所在的进程号和线程当前的状态
两个grep命令分别筛选进程号和线程状态
wc统计线程个数。

ps -eLo pid ,stat | grep 27989输出的结果如下：

图中只截图了部分结果；Sl表示大多数线程都处于空闲状态。

3 Tomcat 连接器框架 : Coyote 【待完善】

3.1 连接器核心功能

一、监听网络端口，接收和响应网络请求。
二、网络字节流处理。

将收到的网络字节流转换成 Tomcat Request 再转成标准的 ServletRequest 给容器，同时将容器传来的 ServletResponse 转成 Tomcat Response 再转成网络字节流。

3.2 连接器模块设计 / 组件构成(Endpoint/Processor/Adapter/ProtocolHandler)

为满足连接器的两个核心功能，我们需要一个通讯端点来监听端口；
需要一个处理器(Connector)来处理网络字节流；
最后，还需要一个适配器将处理后的结果转成容器需要的结构。

对应的源码包路径 org.apache.coyote 。对应的结构图如：

4 Tomcat 容器框架 : Catalina 【待完善】

4.1 容器结构分析(Enginne/Host/Context/Wrapper)

每个 Service 会包含一个容器。

容器由一个引擎(Eginee)可以管理多个虚拟主机(Host)。

每个虚拟主机可以管理多个 Web 应用。

每个 Web 应用会有多个 Servlet 包装器。

Engine、Host、Context 和 Wrapper，四个容器之间属于父子关系。

对应的源码包路径 org.apache.catalina。对应的结构图如下：

4.2 容器请求处理

容器的请求处理过程就是在 Engine、Host、Context 和 Wrapper 这四个容器之间层层调用；

最后在 Servlet 中执行对应的业务逻辑。

各容器都会有一个通道 Pipeline，每个通道上都会有一个 Basic Valve（如StandardEngineValve）， 类似一个闸门用来处理 Request 和 Response 。

其流程图如下：

Y 附件

Y.1 jstack

Y.1.1 jstack -l

C:\Users\johnnyzen>jps

12468

24708 DataServiceBizApplication

29364

29412 Main

30196 Jps

4472 Launcher

19740 jar

C:\Users\johnnyzen>jstack -l 24708

2023-04-07 11:54:06

Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.261-b12 mixed mode):

"nacos-grpc-client-executor-1126" #4688 daemon prio=5 os_prio=0 tid=0x0000014b8685a000 nid=0x4aec waiting on condition [0x000000375e5ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c52eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1135" #4687 daemon prio=5 os_prio=0 tid=0x0000014b86853000 nid=0x5e14 waiting on condition [0x000000375d1fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c4c025a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1112" #4686 daemon prio=5 os_prio=0 tid=0x0000014b86859000 nid=0x67c4 waiting on condition [0x000000375cffe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5b2dfb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1119" #4685 daemon prio=5 os_prio=0 tid=0x0000014b8684b800 nid=0x24ec waiting on condition [0x000000375aefe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5fb6708> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1125" #4684 daemon prio=5 os_prio=0 tid=0x0000014b86851800 nid=0x742c waiting on condition [0x000000375e8ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c52eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1134" #4683 daemon prio=5 os_prio=0 tid=0x0000014b8684f800 nid=0x1f74 waiting on condition [0x000000375c6fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c4c025a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1111" #4682 daemon prio=5 os_prio=0 tid=0x0000014b86857800 nid=0x4c88 waiting on condition [0x000000375adff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5b2dfb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos-grpc-client-executor-1118" #4681 daemon prio=5 os_prio=0 tid=0x0000014b86857000 nid=0x6bd4 waiting on condition [0x000000375e6fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5fb6708> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"RMI TCP Connection(4)-10.33.15.23" #1436 daemon prio=5 os_prio=0 tid=0x0000014bfaa6d800 nid=0x65f0 runnable [0x000000375affe000]

   java.lang.Thread.State: RUNNABLE

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

        at java.net.SocketInputStream.read(SocketInputStream.java:171)

        at java.net.SocketInputStream.read(SocketInputStream.java:141)

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

        - locked <0x000000077cf9a0d0> (a java.io.BufferedInputStream)

        at java.io.FilterInputStream.read(FilterInputStream.java:83)

        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:555)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$957/1082954670.run(Unknown Source)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x000000077ce11940> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.naming.updater.3" #243 daemon prio=5 os_prio=0 tid=0x0000014bfaa6a800 nid=0x6cac waiting on condition [0x00000037619fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d69e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.naming.updater.2" #194 daemon prio=5 os_prio=0 tid=0x0000014b86856000 nid=0x7df4 waiting on condition [0x00000037618ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d69e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"JMX server connection timeout 183" #183 daemon prio=5 os_prio=0 tid=0x0000014bfaa67800 nid=0x7428 in Object.wait() [0x00000037614ff000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)

        - locked <0x00000006c557a8c8> (a [I)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"RMI Scheduler(0)" #182 daemon prio=5 os_prio=0 tid=0x0000014bfaa62000 nid=0x76fc waiting on condition [0x00000037608fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c642a630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"RMI TCP Accept-0" #180 daemon prio=5 os_prio=0 tid=0x0000014b8707d800 nid=0x8720 runnable [0x00000037605fe000]

   java.lang.Thread.State: RUNNABLE

        at java.net.DualStackPlainSocketImpl.accept0(Native Method)

        at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:127)

        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)

        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:189)

        - locked <0x00000006c557a460> (a java.net.SocksSocketImpl)

        at java.net.ServerSocket.implAccept(ServerSocket.java:545)

        at java.net.ServerSocket.accept(ServerSocket.java:513)

        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #160 daemon prio=5 os_prio=0 tid=0x0000014b87073800 nid=0x8688 waiting on condition [0x00000037613fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #158 daemon prio=5 os_prio=0 tid=0x0000014b8706f000 nid=0x7cb8 waiting on condition [0x000000375fdfe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #154 daemon prio=5 os_prio=0 tid=0x0000014b87077800 nid=0x4d30 waiting on condition [0x0000003760bfe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #152 daemon prio=5 os_prio=0 tid=0x0000014b8707a800 nid=0x4014 waiting on condition [0x0000003760afe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #148 daemon prio=5 os_prio=0 tid=0x0000014b8707c800 nid=0x87b4 waiting on condition [0x00000037606ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #146 daemon prio=5 os_prio=0 tid=0x0000014b87076800 nid=0x8870 waiting on condition [0x000000375feff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #141 daemon prio=5 os_prio=0 tid=0x0000014b87078000 nid=0x8014 waiting on condition [0x000000375eaff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #139 daemon prio=5 os_prio=0 tid=0x0000014b87076000 nid=0x59d8 waiting on condition [0x000000375e7fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #136 daemon prio=5 os_prio=0 tid=0x0000014b87071800 nid=0x6fc0 waiting on condition [0x00000037626ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #135 daemon prio=5 os_prio=0 tid=0x0000014b87072000 nid=0x5b50 waiting on condition [0x00000037625fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #134 daemon prio=5 os_prio=0 tid=0x0000014b87074800 nid=0x76f0 waiting on condition [0x000000375e4ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"DestroyJavaVM" #132 prio=5 os_prio=0 tid=0x0000014b87075000 nid=0x66cc waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #118 daemon prio=5 os_prio=0 tid=0x0000014bfaa69000 nid=0x6f4c waiting on condition [0x00000037616fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-8" #114 daemon prio=5 os_prio=0 tid=0x0000014bfaa60000 nid=0x58a4 runnable [0x00000037612fe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4e7d568> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4e7e6a0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4e7e590> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-7" #113 daemon prio=5 os_prio=0 tid=0x0000014bfaa61800 nid=0x3a48 runnable [0x00000037611fe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4e8c2d0> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4e8d408> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4e8d2f8> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.remote.worker" #112 daemon prio=5 os_prio=0 tid=0x0000014bfaa65000 nid=0x862c waiting on condition [0x00000037610fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5f02530> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at com.alibaba.nacos.common.remote.client.RpcClient$2.run(RpcClient.java:295)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5f026e0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.remote.worker" #111 daemon prio=5 os_prio=0 tid=0x0000014bfaa60800 nid=0x5f1c waiting on condition [0x0000003760ffe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5fbebf0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.common.remote.client.RpcClient$3.run(RpcClient.java:317)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5f02908> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.Worker" #110 daemon prio=5 os_prio=0 tid=0x0000014bfaa66800 nid=0x8500 waiting on condition [0x0000003760eff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #109 daemon prio=5 os_prio=0 tid=0x0000014bfaa5f000 nid=0x6e60 waiting on condition [0x0000003760dff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5ec3e98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$4.run(ClientWorker.java:744)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5d8a670> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.Worker" #108 daemon prio=5 os_prio=0 tid=0x0000014bfaa66000 nid=0x8af8 waiting on condition [0x0000003760cfe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d897a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.naming.updater.1" #104 daemon prio=5 os_prio=0 tid=0x0000014b86858800 nid=0x3570 waiting on condition [0x00000037609ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d69e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-6" #99 daemon prio=5 os_prio=0 tid=0x0000014b8684e800 nid=0x54a4 runnable [0x00000037604fe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4eb2ef8> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4eb4030> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4eb3f20> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-5" #98 daemon prio=5 os_prio=0 tid=0x0000014b86855800 nid=0x6308 runnable [0x00000037603fe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4eb9c88> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4ed2ca0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4ed2b90> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.remote.worker" #97 daemon prio=5 os_prio=0 tid=0x0000014b8684e000 nid=0x8990 waiting on condition [0x00000037602ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5dd30b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.common.remote.client.RpcClient$3.run(RpcClient.java:317)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5c80dc0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.remote.worker" #96 daemon prio=5 os_prio=0 tid=0x0000014b86854800 nid=0x726c waiting on condition [0x00000037601fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at com.alibaba.nacos.common.remote.client.RpcClient$2.run(RpcClient.java:295)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5c80b78> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.Worker" #95 daemon prio=5 os_prio=0 tid=0x0000014b86854000 nid=0x7ab8 waiting on condition [0x00000037600ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #94 daemon prio=5 os_prio=0 tid=0x0000014b8684d000 nid=0x6c64 waiting on condition [0x000000375fffe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c801e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$4.run(ClientWorker.java:744)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5c7ff88> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.Worker" #93 daemon prio=5 os_prio=0 tid=0x0000014b8684c800 nid=0x6b70 waiting on condition [0x000000375e3ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c7f060> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-Acceptor" #89 daemon prio=5 os_prio=0 tid=0x0000014b86851000 nid=0x7f64 runnable [0x000000375fbfe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252)

        - locked <0x00000006c63c5a00> (a java.lang.Object)

        at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:574)

        at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:80)

        at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-ClientPoller" #88 daemon prio=5 os_prio=0 tid=0x0000014b8684b000 nid=0x5d98 runnable [0x000000375faff000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c62c1e68> (a sun.nio.ch.Util$3)

        - locked <0x00000006c62c1e58> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c62c1c18> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:816)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-10" #87 daemon prio=5 os_prio=0 tid=0x0000014bfda04800 nid=0x4ff0 waiting on condition [0x000000375f9fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-9" #86 daemon prio=5 os_prio=0 tid=0x0000014bfda03000 nid=0x7ae8 waiting on condition [0x000000375f8ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-8" #85 daemon prio=5 os_prio=0 tid=0x0000014bfda05000 nid=0x7cc0 waiting on condition [0x000000375f7fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-7" #84 daemon prio=5 os_prio=0 tid=0x0000014bfda06000 nid=0x8484 waiting on condition [0x000000375f6ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-6" #83 daemon prio=5 os_prio=0 tid=0x0000014bfa33e800 nid=0x7dd4 waiting on condition [0x000000375f5ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-5" #82 daemon prio=5 os_prio=0 tid=0x0000014bfa33e000 nid=0x1eb0 waiting on condition [0x000000375f4fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-4" #81 daemon prio=5 os_prio=0 tid=0x0000014bfa33c800 nid=0x84d8 waiting on condition [0x000000375f3fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-3" #80 daemon prio=5 os_prio=0 tid=0x0000014bfa342800 nid=0x63bc waiting on condition [0x000000375f2fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-2" #79 daemon prio=5 os_prio=0 tid=0x0000014bfa33b800 nid=0x615c waiting on condition [0x000000375f1fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-exec-1" #78 daemon prio=5 os_prio=0 tid=0x0000014bfa340000 nid=0x7c98 waiting on condition [0x000000375f0ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c6340cb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)

        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"http-nio-9527-BlockPoller" #77 daemon prio=5 os_prio=0 tid=0x0000014bfa33d000 nid=0x78a8 runnable [0x000000375efff000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c626b980> (a sun.nio.ch.Util$3)

        - locked <0x00000006c626b970> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c626b810> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:331)

   Locked ownable synchronizers:

        - None

"Nacos-Watch-Task-Scheduler-1" #76 prio=5 os_prio=0 tid=0x0000014bfa341800 nid=0xb24 waiting on condition [0x000000375eeff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5bda6d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.naming.updater.0" #74 daemon prio=5 os_prio=0 tid=0x0000014b86662800 nid=0x65f8 waiting on condition [0x000000375ecff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d69e58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-4" #69 daemon prio=5 os_prio=0 tid=0x0000014b8665d800 nid=0x5ea0 runnable [0x000000375dcfe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4ed88f8> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4ed9a30> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4ed9920> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-3" #68 daemon prio=5 os_prio=0 tid=0x0000014b8665f800 nid=0x6ffc runnable [0x000000375dbfe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4eff5b8> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4f006f0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4f005e0> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.remote.worker" #67 daemon prio=5 os_prio=0 tid=0x0000014b86661000 nid=0x4638 waiting on condition [0x000000375dafe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5ae1768> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at com.alibaba.nacos.common.remote.client.RpcClient$2.run(RpcClient.java:295)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5ae1918> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.remote.worker" #66 daemon prio=5 os_prio=0 tid=0x0000014b8665c000 nid=0x26ac waiting on condition [0x000000375d9fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c23e80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.common.remote.client.RpcClient$3.run(RpcClient.java:317)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5ae1b60> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.naming.grpc.redo.0" #65 daemon prio=5 os_prio=0 tid=0x0000014b86653800 nid=0x17fc waiting on condition [0x000000375d8ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5d89030> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.naming.push.receiver" #64 daemon prio=5 os_prio=0 tid=0x0000014b86656000 nid=0x4564 runnable [0x000000375d7fe000]

   java.lang.Thread.State: RUNNABLE

        at java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)

        at java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:124)

        - locked <0x00000006c5e4a260> (a java.net.DualStackPlainDatagramSocketImpl)

        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)

        - locked <0x00000006c5e4a260> (a java.net.DualStackPlainDatagramSocketImpl)

        at java.net.DatagramSocket.receive(DatagramSocket.java:812)

        - locked <0x00000006c65cda38> (a java.net.DatagramPacket)

        - locked <0x00000006c5e4a220> (a java.net.DatagramSocket)

        at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:102)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c5e49da8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.naming.security" #63 daemon prio=5 os_prio=0 tid=0x0000014b8665a800 nid=0x4c10 waiting on condition [0x000000375d6ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5e4a728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.naming.failover" #61 daemon prio=5 os_prio=0 tid=0x0000014b86658000 nid=0x628c waiting on condition [0x000000375d5ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5c65170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent" #60 daemon prio=5 os_prio=0 tid=0x0000014b86659800 nid=0x8538 waiting on condition [0x000000375d4ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5bd9c68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)

        at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:117)

        at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)

   Locked ownable synchronizers:

        - None

"spring.cloud.inetutils" #59 daemon prio=5 os_prio=0 tid=0x0000014b86657800 nid=0x61ec waiting on condition [0x000000375d3fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5e4aa28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #58 daemon prio=5 os_prio=0 tid=0x0000014b86656800 nid=0x81b0 waiting on condition [0x000000375d2ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"HikariPool-1 housekeeper" #51 daemon prio=5 os_prio=0 tid=0x0000014bfa347000 nid=0x8198 waiting on condition [0x000000375e2fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c5aacc70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #50 daemon prio=5 os_prio=0 tid=0x0000014bfa33f800 nid=0x7f14 waiting on condition [0x000000375e1ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"mysql-cj-abandoned-connection-cleanup" #49 daemon prio=5 os_prio=0 tid=0x0000014bfa345800 nid=0x774c in Object.wait() [0x000000375e0ff000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000006c65cf720> (a java.lang.ref.ReferenceQueue$Lock)

        at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:91)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c65cf7b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"container-0" #47 prio=5 os_prio=0 tid=0x0000014bfa344800 nid=0x872c waiting on condition [0x000000375dffe000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

        at java.lang.Thread.sleep(Native Method)

        at org.apache.catalina.core.StandardServer.await(StandardServer.java:570)

        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:197)

   Locked ownable synchronizers:

        - None

"Catalina-utility-2" #46 prio=1 os_prio=-2 tid=0x0000014bfa346000 nid=0x6068 waiting on condition [0x000000375defe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c63c5468> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"Catalina-utility-1" #45 prio=1 os_prio=-2 tid=0x0000014bfa344000 nid=0x7914 waiting on condition [0x000000375c7fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c63c5468> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #44 daemon prio=5 os_prio=0 tid=0x0000014bfa341000 nid=0x1078 waiting on condition [0x000000375ddfe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-2" #29 daemon prio=5 os_prio=0 tid=0x0000014bfda03800 nid=0x48bc runnable [0x000000375d0fe000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c4f15e38> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c4f16f10> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c4f15d58> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"grpc-nio-worker-ELG-1-1" #27 daemon prio=5 os_prio=0 tid=0x0000014bfc015000 nid=0x7820 runnable [0x000000375ceff000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        - locked <0x00000006c5028d78> (a com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySet)

        - locked <0x00000006c5029de0> (a java.util.Collections$UnmodifiableSet)

        - locked <0x00000006c5028ca8> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:806)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.remote.worker" #26 daemon prio=5 os_prio=0 tid=0x0000014bfab8d000 nid=0x8898 waiting on condition [0x000000375cdff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c4c01fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.common.remote.client.RpcClient$3.run(RpcClient.java:317)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c4bf88a8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.remote.worker" #25 daemon prio=5 os_prio=0 tid=0x0000014bfab8c000 nid=0x60d4 waiting on condition [0x000000375ccfe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c4bf8458> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

        at com.alibaba.nacos.common.remote.client.RpcClient$2.run(RpcClient.java:295)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c4bf8660> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"nacos.publisher-com.alibaba.nacos.common.notify.SlowEvent" #23 daemon prio=5 os_prio=0 tid=0x0000014bfa832800 nid=0x6cf4 waiting on condition [0x000000375cbfe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c4bf8248> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)

        at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:117)

        at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #21 daemon prio=5 os_prio=0 tid=0x0000014bfc2c2000 nid=0x7db4 waiting on condition [0x000000375cafe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #20 daemon prio=5 os_prio=0 tid=0x0000014bfc2c1000 nid=0x4d10 waiting on condition [0x000000375c9fe000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c5388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"com.alibaba.nacos.client.Worker" #19 daemon prio=5 os_prio=0 tid=0x0000014bf99a8800 nid=0x3a70 waiting on condition [0x000000375c8fe000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c49c97d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)

        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$4.run(ClientWorker.java:744)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - <0x00000006c4ab1178> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"com.alibaba.nacos.client.identify.watcher.0" #16 daemon prio=5 os_prio=0 tid=0x0000014bfe041800 nid=0x23dc runnable [0x000000375c5ff000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00000006c45bcec8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

"Service Thread" #11 daemon prio=9 os_prio=0 tid=0x0000014bfa363800 nid=0x3b84 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"C1 CompilerThread3" #10 daemon prio=9 os_prio=2 tid=0x0000014bfa298000 nid=0xd1c waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"C2 CompilerThread2" #9 daemon prio=9 os_prio=2 tid=0x0000014bfa297800 nid=0x88f0 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"C2 CompilerThread1" #8 daemon prio=9 os_prio=2 tid=0x0000014bfa294800 nid=0x2b6c waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"C2 CompilerThread0" #7 daemon prio=9 os_prio=2 tid=0x0000014bfa292000 nid=0x16f8 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"Monitor Ctrl-Break" #6 daemon prio=5 os_prio=0 tid=0x0000014bfa290800 nid=0x9e4 runnable [0x000000375beff000]

   java.lang.Thread.State: RUNNABLE

        at java.net.SocketInputStream.socketRead0(Native Method)

        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

        at java.net.SocketInputStream.read(SocketInputStream.java:171)

        at java.net.SocketInputStream.read(SocketInputStream.java:141)

        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)

        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)

        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)

        - locked <0x00000006c4023a68> (a java.io.InputStreamReader)

        at java.io.InputStreamReader.read(InputStreamReader.java:184)

        at java.io.BufferedReader.fill(BufferedReader.java:161)

        at java.io.BufferedReader.readLine(BufferedReader.java:324)

        - locked <0x00000006c4023a68> (a java.io.InputStreamReader)

        at java.io.BufferedReader.readLine(BufferedReader.java:389)

        at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:55)

   Locked ownable synchronizers:

        - None

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000014bf7b87000 nid=0x7f9c waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000014bf7b86000 nid=0x8548 runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:

        - None

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x0000014bf7778800 nid=0x6b78 in Object.wait() [0x000000375bbff000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

        - locked <0x00000006c402cdc8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

   Locked ownable synchronizers:

        - None

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000014bf7b58000 nid=0x2984 in Object.wait() [0x000000375b1ff000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:502)

        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)

        - locked <0x00000006c402cf80> (a java.lang.ref.Reference$Lock)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:

        - None

"VM Thread" os_prio=2 tid=0x0000014bf7b31800 nid=0x6938 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000014bdcf10800 nid=0x8a54 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000014bdcf12000 nid=0x8384 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000014bdcf13000 nid=0x5664 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000014bdcf14800 nid=0x36fc runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000014bdcf16800 nid=0x6ff0 runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000014bdcf17800 nid=0x75d0 runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x0000014bdcf1a800 nid=0x754c runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x0000014bdcf1c800 nid=0x8110 runnable

"VM Periodic Task Thread" os_prio=2 tid=0x0000014bfa53a000 nid=0x6358 waiting on condition

JNI global references: 2003

C:\Users\johnnyzen>

U 推荐文献

Apache Tomcat 8 Configuration Reference - Apache Tomcat
《How Tomcat Works》
《深入分析Java Web技术内幕》
Tomcat 6 ---- Realm域管理 - 博客园
Tomcat Port 8009 与AJP13协议 - 163博客
使用Jasper引擎解析JSP - 博客园

X 参考文献

tomcat调优之监控连接数和连接池 - CSDN
Springboot 内嵌Tomcat 的http连接池与thread的关系 - 博客园
Tomcat原理 - Zhihu/分布式系统架构：技术栈详解与快速进阶
详解Tomcat 配置文件server.xml - 博客园/编程迷思
tomcat中server.xml配置详解 - 博客园/starhu
阿里二面：你知道 Tomcat 的工作原理么？ - Zhihu/Java编程开发者

[Web Server]Tomcat调优之工作原理、线程池/连接池的相关教程结束。