Fork me on GitHub

Trouble Shooting —— CAS Server集群环境下TGC验证问题排查,需要开启会话保持

问题现象

CAS部署结构:

两台cas server通过nginx做负载均衡,两个cas serverticket registry配置的jpa方式,指向同一个库。两个cas servertomcat做了TomcatRedisSessionManager,使用redis集中存储session

目前的现象:

页面上请求cas登录地址,登录过后频繁刷新登录页面,有时返回已登录,有时返回未登录,当返回未登录时去后台查看日志发现有如下错误,验证cookie发现请求的源IP与第一次访问的源IP不一致。这个很明显是cas集群环境下的问题。

2018-03-16 10:02:44,418 DEBUG [org.apereo.cas.web.support.TGCCookieRetrievingCookieGenerator] - <Invalid cookie. Required remote address does not match ${ip}>
java.lang.IllegalStateException: Invalid cookie. Required remote address does not match ${ip}
	at org.apereo.cas.web.support.DefaultCasCookieValueManager.obtainCookieValue(DefaultCasCookieValueManager.java:84) ~[cas-server-support-cookie-5.0.4.jar:5.0.4]
	at org.apereo.cas.web.support.CookieRetrievingCookieGenerator.retrieveCookieValue(CookieRetrievingCookieGenerator.java:93) ~[cas-server-support-cookie-5.0.4.jar:5.0.4]
	at org.apereo.cas.web.support.CookieRetrievingCookieGenerator$$FastClassBySpringCGLIB$$25dba342.invoke(<generated>) ~[cas-server-support-cookie-5.0.4.jar:5.0.4]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apereo.cas.web.support.CookieRetrievingCookieGenerator$$EnhancerBySpringCGLIB$$10d36968.retrieveCookieValue(<generated>) ~[cas-server-support-cookie-5.0.4.jar:5.0.4]
	at org.apereo.cas.logging.web.ThreadContextMDCServletFilter.doFilter(ThreadContextMDCServletFilter.java:83) ~[cas-server-core-logging-5.0.4.jar:5.0.4]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[spring-boot-actuator-1.4.2.RELEASE.jar:1.4.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117) ~[spring-boot-1.4.2.RELEASE.jar:1.4.2.RELEASE]
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61) ~[spring-boot-1.4.2.RELEASE.jar:1.4.2.RELEASE]
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92) ~[spring-boot-1.4.2.RELEASE.jar:1.4.2.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.4.RELEASE.jar:4.3.4.RELEASE]
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110) ~[spring-boot-1.4.2.RELEASE.jar:1.4.2.RELEASE]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) ~[log4j-web-2.6.2.jar:2.6.2]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) ~[catalina.jar:7.0.85]
	at com.r.tomcat.session.management.RequestSessionHandlerValve.invoke(RequestSessionHandlerValve.java:30) ~[TomcatRedisSessionManager-1.0.jar:?]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) ~[catalina.jar:7.0.85]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) ~[catalina.jar:7.0.85]
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) ~[catalina.jar:7.0.85]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) ~[catalina.jar:7.0.85]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) ~[catalina.jar:7.0.85]
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115) ~[tomcat-coyote.jar:7.0.85]
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) ~[tomcat-coyote.jar:7.0.85]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775) ~[tomcat-coyote.jar:7.0.85]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734) ~[tomcat-coyote.jar:7.0.85]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-coyote.jar:7.0.85]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

网上查询资料:google group,相同的问题,但是没有看到具体的解决方法。

看到的tomcat RemoteIpValue也只是tomcat请求ip限制的方法,跟我们要的不匹配

根据异常查看CAS代码,如下:

 public String obtainCookieValue(Cookie cookie, HttpServletRequest request)
  {
    String cookieValue = (String)this.cipherExecutor.decode(cookie.getValue());
    LOGGER.debug("Decoded cookie value is [{}]", cookieValue);
    if (StringUtils.isBlank(cookieValue))
    {
      LOGGER.debug("Retrieved decoded cookie value is blank. Failed to decode cookie [{}]", cookie.getName());
      return null;
    }
    String[] cookieParts = cookieValue.split(String.valueOf('@'));
    if (cookieParts.length != 3) {
      throw new IllegalStateException("Invalid cookie. Required fields are missing");
    }
    String value = cookieParts[0];
    String remoteAddr = cookieParts[1];
    String userAgent = cookieParts[2];
    if ((StringUtils.isBlank(value)) || (StringUtils.isBlank(remoteAddr)) || 
      (StringUtils.isBlank(userAgent))) {
      throw new IllegalStateException("Invalid cookie. Required fields are empty");
    }
    if (!remoteAddr.equals(request.getRemoteAddr())) {
      throw new IllegalStateException("Invalid cookie. Required remote address does not match " + request.getRemoteAddr());
    }
    String agent = WebUtils.getHttpServletRequestUserAgent(request);
    if (!userAgent.equals(agent)) {
      throw new IllegalStateException("Invalid cookie. Required user-agent does not match " + agent);
    }
    return value;
  }

TGC中包含了user-agent信息,会根据requestuser-agent去跟decode后的cookie中的user-agent对比,而且这个验证是在cas 4.1版本就已经加了这个验证信息了,如果我们修改源码去掉这个user-agent验证可能还会引发其他问题。

解决方案

因此我们选用负载均衡的保持会话来解决这个问题了。

  1. 如果使用的是阿里云的SLB需要开启会话保持的选项。
  2. 如果使用nginx需要在upstream中增加ip_hash保持会话。

这样就可以让相同的客户端ip将会话永远路由到相同的一台后端cas server上去。

经过验证解决了上述的问题。

所以这里需要说明一下,在对cas server做集群实现无状态化,需要注意一下几点:

  1. casticket需要做到集中存储,可以使用redisjpa、或者其他方式,这个官方文章上有详细介绍:ticket-registry
  2. cassession信息需要做到集中存储,如果使用的是tomcat可以使用TomcatRedisSessionMananger插件来通过redis做session集中存储。
  3. 还有一个就是上面遇到的问题,客户端cookie信息:TGCTGC采用cookie方式存在客户端,因此需要开启会话保持,使得相同客户端每次都会被路由到同一个cas server上去做TGC验证。
  4. 最后一个就是需要接入ssoclient应用端的session信息也需要做集中存储,因此cas server会和client进行通信去验证ticket,验证完后会生成信息并存储到sesson中,因此也需要使用TomcatRedisSessionMananger插件来通过redissession集中存储。

世界和平、Keep Real!

Comments