영이네

How to detect leaked datasource connections using the cached connection manager (CCM) debug facility in JBoss EAP

https://access.redhat.com/solutions/309913

JBoss EAP 6 Native 설정

JBoss 또는 Tomcat에 Native 모듈을 설정할 경우 사용자의 요청을 받아들이는 OS의 Native 영역에 Non blocking 방식으로 접근이 가능해진다.

그래서 처리량이 증가하여 성능 향상이 되거나 Thread의 Wait 현상이 완화된다.

추가로 JBoss EAP 7은 XNIO를 사용하므로 추가 설정 없이 사용가능하다

1. Native 모듈 다운로드 - JBoss가 설치된 OS 환경에 맞는 모듈 다운로드

- http://access.redhat.com (다운받기 위해서는 서브스크립션 필요)

2. Native 모듈 설치(리눅스 예제)

2.1. 모듈 복사

모듈을 압축 해제하고 제이보스 엔진의 modules 이하 동일한 경로에 해당 디렉터리와 모듈 복사 

modules/system/layers/base/org/hornetq/main/lib/linux-x86_64

modules/system/layers/base/org/jboss/as/web/main/lib/linux-x86_64

] cp -r modules/system/layers/base/org/hornetq/main/lib/linux-x86_64 $JBOSS_HOME/modules/system/layers/base/org/hornetq/main/lib

] cp -r modules/system/layers/base/org/jboss/as/web/main/lib/linux-x86_64 $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/lib

2.2. module.xml 수정

추가한 Native 모듈을 사용할 수 있도록 module.xml의 resource-root에 모듈 경로 지정

] vi $JBOSS_HOME/modules/system/layers/base/org/hornetq/main/module.xml

] vi $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml

<resources>

…

<resource-root path="lib/linux-x86_64"/>

</resources>

3. standalone/domain.xml 수정 - web subsystem에 native 사용 활성화

JBoss EAP 6에서는 기본으로 Native 모듈을 사용안함으로 설정되어 있음

] vi ${jboss.server.base.dir}/configuration/stanalone.xml or ${jboss.domain.base.dir}/configuration/domain.xml

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtualserver="default-host" native="true">

4. 재기동

JBoss EAP 6 Native 설정

JBoss 또는 Tomcat에 Native 모듈을 설정할 경우 사용자의 요청을 받아들이는 OS의 Native 영역에 Non blocking 방식으로 접근이 가능해진다.

그래서 처리량이 증가하여 성능 향상이 되거나 Thread의 Wait 현상이 완화된다.

추가로 JBoss EAP 7은 XNIO를 사용하므로 추가 설정 없이 사용가능하다

1. Native 모듈 다운로드 - JBoss가 설치된 OS 환경에 맞는 모듈 다운로드

- http://access.redhat.com (다운받기 위해서는 서브스크립션 필요)

2. Native 모듈 설치(리눅스 예제)

2.1. 모듈 복사

모듈을 압축 해제하고 제이보스 엔진의 modules 이하 동일한 경로에 해당 디렉터리와 모듈 복사 

modules/system/layers/base/org/hornetq/main/lib/linux-x86_64

modules/system/layers/base/org/jboss/as/web/main/lib/linux-x86_64

] cp -r modules/system/layers/base/org/hornetq/main/lib/linux-x86_64 $JBOSS_HOME/modules/system/layers/base/org/hornetq/main/lib

] cp -r modules/system/layers/base/org/jboss/as/web/main/lib/linux-x86_64 $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/lib

2.2. module.xml 수정

추가한 Native 모듈을 사용할 수 있도록 module.xml의 resource-root에 모듈 경로 지정

] vi $JBOSS_HOME/modules/system/layers/base/org/hornetq/main/module.xml

] vi $JBOSS_HOME/modules/system/layers/base/org/jboss/as/web/main/module.xml

<resources>

…

<resource-root path="lib/linux-x86_64"/>

</resources>

3. standalone/domain.xml 수정 - web subsystem에 native 사용 활성화

JBoss EAP 6에서는 기본으로 Native 모듈을 사용안함으로 설정되어 있음

] vi ${jboss.server.base.dir}/configuration/stanalone.xml or ${jboss.domain.base.dir}/configuration/domain.xml

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtualserver="default-host" native="true">

4. 재기동

1. keystore/vault 파일을 저장할 디렉터리 생성

web1]$ mkdir -p $JBOSS_HOME/vault

2. keystore.sh 실행하여 keystore 생성

- 스크립트1. keysotre.sh 확인(맨아래)

web1]$ ./keystore.sh [aliasname] [storepass] [keypass] [validity(in day)]

3. vault.sh 실행하여 vault 생성

- 스크립트2. vault.sh 확인(맨아래)

web1]$ ./vault.sh [aliasname] [sotrepass] [password]

4. 출력된 결과를 standalone.xml / host-slave.xml / domain.xml에 등록

Please make note of the following:
********************************************
Vault Block:vb
Attribute Name:password
Configuration should be done as follows:
VAULT::vb::password::1
********************************************
WFLYSEC0048: Vault Configuration in WildFly configuration file:
********************************************
...
</extensions>
<vault>
  <vault-option name="KEYSTORE_URL" value="$JBOSS_HOME/vault/vault.keystore"/>
  <vault-option name="KEYSTORE_PASSWORD" value="MASK-5dOaAVafCSd"/>
  <vault-option name="KEYSTORE_ALIAS" value="vault"/>
  <vault-option name="SALT" value="1234abcd"/>
  <vault-option name="ITERATION_COUNT" value="120"/>
  <vault-option name="ENC_FILE_DIR" value="$JBOSS_HOME/vault/"/>
</vault>
<management>
...
********************************************

4.1. standalone 모드

web1]$ vi standalone.xml

...
</extensions>
<vault>

<vault-option name="KEYSTORE_URL" value="$JBOSS_HOME/vault/vault.keystore"/>
<vault-option name="KEYSTORE_PASSWORD" value="MASK-5dOaAVafCSd"/>
<vault-option name="KEYSTORE_ALIAS" value="vault"/>
<vault-option name="SALT" value="1234abcd"/>
<vault-option name="ITERATION_COUNT" value="120"/>
<vault-option name="ENC_FILE_DIR" value="$JBOSS_HOME/vault/"/>

</vault>
<management>
...

<datasource pool-name="OracleDS" ...>

<security>

<user-name>jboss</user-name>
<password>${VAULT::vb::password::1}</password>

</security>

</datasource>

4.2. domain 모드

web1]$ vi host-slave.xml

...
</extensions>
<vault>

<vault-option name="KEYSTORE_URL" value="$JBOSS_HOME/vault/vault.keystore"/>
<vault-option name="KEYSTORE_PASSWORD" value="MASK-5dOaAVafCSd"/>
<vault-option name="KEYSTORE_ALIAS" value="vault"/>
<vault-option name="SALT" value="1234abcd"/>
<vault-option name="ITERATION_COUNT" value="120"/>
<vault-option name="ENC_FILE_DIR" value="$JBOSS_HOME/vault/"/>

</vault>
<management>
...

web1]$ vi domain.xml

<datasource pool-name="OracleDS" ...>

<security>

<user-name>jboss</user-name>
<password>${VAULT::vb::password::1}</password>

</security>

</datasource>

5. 재기동

# 스크립트1. keystore.sh

#!/bin/sh

. ./env.sh

if[ \( -n "$1" \) -a \( -n "$2" \) -a \( -n "$3" \) -a \( -n "$4" \) ]
then

$JAVA_HOME/bin/keytool -genseckey -alias $1 -storetype jceks -keyalg AES -keysize 128 -storepass $2 -keypass $3 -validity $4 -keystore $JBOSS_HOME/vault/vault.keystore
echo "Check: $JBOSS_HOME/vault/vault.keystore"
exit;

fi

echo "./keystore.sh [aliasname] [storepass] [keypass] [validity(in day)]"

exit;

# 스크립트2. vault.sh

#!/bin/sh

. ./env.sh

if[ \( -n "$1" \) -a \( -n "$2" \) -a \( -n "$3" \) ]
then

$JBOSS_HOME/bin/vault.sh --keystore $JBOSS_HOME/vault/vault.keystore --alias $1 --keystore-password $2 --vault-block vb --attribute password --sec-attr $3 --enc-dir $EAP_HOME/vault/ --iteration 120 --salt 1234abcd

echo "Check: $JBOSS_HOME/vault/VAULT.dat"

exit;

fi

echo "./vault.sh [aliasname] [storepass] [password]"

exit;

# 참고 URL : https://access.redhat.com/solutions/2790371

JBoss EAP 6 - StuckThread 감지 설정

제이보스에서 지정한 시간(초)동안 쓰레드가 수행될 경우 로그에 경고 메시지를 발생시키며

해당 쓰레드의 stacktrace를 로그에 기록함

설정파일: standalone.xml / domain.xml

설정내용:

<subsystem xmlns="urn:jboss:domain:web:2.2" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="true">

<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" executor="ajp-thread-pool"/>

<virtual-server name="default-host" enable-welcome-root="false">

<alias name="localhost"/>
<alias name="example.com"/>
<access-log pattern='%h %l %u %t %r %s %b %{Referer}i %{User-Agent}i %s %T'>

<directory path="access" relative-to="jboss.server.log.dir"/>

</access-log>

</virtual-server>
<valve name="StuckThreadValve" module="org.jboss.as.web" class-name="org.apache.catalina.valves.StuckThreadDetectionValve">

<param param-name="threshold" param-value="10"/>

</valve>

</subsystem>

로그내용:

Stuckthread 발생시

13:52:47,511 WARN  [org.apache.catalina.valves] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBWEB001111: Thread http-192.168.56.102:8180-1 (id=121) has been active for 12548 milliseconds (since Wed Nov 01 13:52:34 KST 2017) to serve the same request for http://192.168.56.102:8180/index.jsp and may be stuck (configured threshold for this StuckThreadDetectionValve is 10 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable
        at java.lang.Thread.sleep(Native Method) [rt.jar:1.7.0_80]
        at org.apache.jsp.index_jsp._jspService(index_jsp.java:59)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:365) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:246) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:134) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
        at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
        at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:95) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
        at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:151) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:132) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:611) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2082) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]

Stuckthread 완료시
13:53:07,540 WARN  [org.apache.catalina.valves] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBWEB001112: Thread http-192.168.56.102:8180-1 (id=121) was previously reported to be stuck but has completed. It was active for approximately 30398 milliseconds. There is/are still 0 thread(s) that are monitored by this Valve and may be stuck. 

JBoss EAP 7 - StuckThread 감지 설정

제이보스에서 지정한 시간(초)동안 쓰레드가 수행될 경우 로그에 경고 메시지를 발생시키며

해당 쓰레드의 stacktrace를 로그에 기록함

설정파일: standalone.xml / domain.xml

설정내용:

<subsystem xmlns="urn:jboss:domain:undertow:3.1" instance-id="${jboss.server.name}" statistics-enabled="true">

<buffer-cache name="default"/>
<server name="default-server">

<ajp-listener name="ajp" socket-binding="ajp" worker="ajp-worker"/>
<http-listener name="default" socket-binding="http" redirect-socket="https" max-parameters="10000"/>

<host name="default-host" alias="localhost">

<location name="/" handler="welcome-content"/>
<access-log prefix="access." directory="${jboss.server.log.dir}/access" pattern="%h %l %u %t %r %s %b %{i,Referer} %{i,User-Agent} %s %T"/>
<filter-ref name="server-header"/>
<filter-ref name="x-powered-by-header"/>
<filter-ref name="stuck" />

</host>

</server>
<servlet-container name="default" default-encoding="UTF-8">

<jsp-config/>
<websockets/>

</servlet-container>
<handlers>

<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

</handlers>
<filters>

<response-header name="server-header" header-name="Server" header-value="JBoss-EAP/7"/>
<response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
<expression-filter name="stuck" expression="blocking; stuck-thread-detector(10)"/>

</filters>

</subsystem>

로그내용:

Stuckthread 발생시

2017-11-01 14:07:42,156 WARN  [io.undertow.request] (default I/O-3) UT005072: Thread default task-1 (id=143) has been active for 10020 milliseconds (since Wed Nov 01 14:07:32 KST 2017) to serve the same request for /index.jsp and may be stuck (configured threshold for this StuckThreadDetectionValve is 10 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable
        at java.lang.Thread.sleep(Native Method)
        at org.apache.jsp.index_jsp._jspService(index_jsp.java:96)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:402)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:346)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
        at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:62)
        at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:189)
        at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:39)
        at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:65)
        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
        at io.undertow.server.handlers.accesslog.AccessLogHandler.handleRequest(AccessLogHandler.java:138)
        at org.wildfly.extension.undertow.Host$OptionsHandler.handleRequest(Host.java:285)
        at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:65)
        at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90)
        at io.undertow.server.handlers.SetHeaderHandler.handleRequest(SetHeaderHandler.java:90)
        at io.undertow.predicate.PredicatesHandler.handleRequest(PredicatesHandler.java:110)
        at io.undertow.server.handlers.StuckThreadDetectionHandler.handleRequest(StuckThreadDetectionHandler.java:168)
        at io.undertow.predicate.PredicatesHandler.handleRequest(PredicatesHandler.java:93)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:324)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:803)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

Stuckthread 완료시
2017-11-01 14:08:03,193 WARN  [io.undertow.request] (default I/O-3) UT005073: Thread default task-1 (id=143) was previously reported to be stuck but has completed. It was active for approximately 30523 milliseconds. There is/are still 0 thread(s) that are monitored by this Valve and may be stuck.