[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests

Hi Rich,

We noticed that with the most recent jdk and tomcat downloads (1.7u55 
and 7u53, respectively), our Windows machines had an old SSL, as 
evidenced here:

Apr 21, 2014 2:34:22 PM org.apache.catalina.core.AprLifecycleListener 
initializeSSL
INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)

Clearly, the OpenSSL library predates the Heartbleed fix.  We asked Jen about 
this, and her advice was to keep the APRLifecycleListener commented out (as we 
have on our production machines).  This avoids the vulnerability until Apache 
Tomcat updates.

-Lansing

On 4/22/2014 9:24 AM, Signell, Richard wrote:
> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
> due to flood of DNS requests
>
> Are there things we should do to respond to Heartbleed on public
> THREDDS servers?
>
> For folks running public IPython notebook servers, we received this
> advice:"If you run a public #IPython notebook server over HTTPS, shut
> it down, upgrade OpenSSL, regenerate your TLS certificate & restart."
>
> Does the same apply to tomcat/thredds?
>
> -Rich
>
> On Mon, Apr 21, 2014 at 5:12 PM, Unidata THREDDS Support
> <address@hidden> wrote:
>> Hi Rich, all,
>>
>> Nothing is jumping out at me. Do you have the access log and 
>> threddsServlet.log files for the time period when the DNS requests were 
>> being generated?
>>
>> The only /tmp file listed below that looks legit (i.e., likely produced by 
>> the TDS) is the hsperfdata_tomcat/ directory (more below [1]). The only 
>> other one that seems like it might be OK is the httpdlog file. But even that 
>> seems a bit odd unless you are running an Apache server as the tomcat user. 
>> Is the tomcat user only used to run Tomcat instances running TDS?
>>
>>
>> [1] The hsperfdata_tomcat/ directory is the only one from the file listing 
>> below that we have on our main server. It, according to [2], is created by 
>> Java and used to allow/improve monitoring capabilities with jconsole and the 
>> like.
>>
>> [2] 
>> http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
>>
>>> Thredds guys,
>>>
>>> UMASS shutdown their production tomcat/thredds and disabled the tomcat
>>> user on Saturday, which of course is causing an interuption in ocean
>>> forecast products in New England used by the US Coast Guard, US IOOS
>>> and the local Weather Service Offices.
>>>
>>> Here is there message about why they shut it down.
>>>
>>> Any ideas about what was happening and how to get this back up and running?
>>>
>>>  From Kent Gardner at UMASSD:
>>>
>>> It appears that the SMAST host system that is running Thredds was
>>> generating a storm of DNS requests to our campus name server. When
>>> Mike shut Thredds down and disabled the tomcat account the storm
>>> stopped.
>>>
>>> I can think of no legitimate reason why Thredds would be doing this.
>>> The only thing that remotely comes to mind would be someone trying to
>>> look up IP numbers in a log file to get the host name for
>>> informational purposes. Has anyone come across this behavior before in
>>> Thredds/Tomcat?
>>>
>>> Also looking in /tmp we see the following:
>>>
>>> ls -al /tmp|grep tomcat
>>>
>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
>>>
>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
>>>
>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
>>>
>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
>>>
>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
>>>
>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 gates.lock
>>>
>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
>>> hsperfdata_tomcat
>>>
>>> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
>>>
>>> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 notify.file
>>>
>>>
>>> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
>>> anyone know if any of these files are legitimate?
>>>
>>> As far a game plan goes I will need to confer with Mike. At the very
>>> least we need to scan for and delete all suspicious files, and change
>>> the password on the tomcat account. After that we start things up and
>>> monitor the network traffic. "
>>>
>>> Thanks,
>>> Rich
>>>
>>> --
>>> Dr. Richard P. Signell
>>
>> Ticket Details
>> ===================
>> Ticket ID: IXX-362335
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>



Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open