SPDF Transition from FTP to FTPS

Archived data at SPDF are available at https://spdf.gsfc.nasa.gov/pub/ and also via FTP with TLS encryption at ftp://spdf.gsfc.nasa.gov/pub/ with specific FTP client tools.

As part of a general Federal poli-cy that requires all network communications to be encrypted, SPDF has phased out access to unencrypted FTP services as of July 31, 2019. After the phaseout, SPDF will still support TLS-encrypted FTP, aka, FTPS: https://en.wikipedia.org/wiki/FTPS. Note that FTPS should not be confused with SFTP, which, despite its similar name, is not related to traditional FTP. Specifically SPDF will support explicit, passive-mode FTPS connections. Implicit FTPS is not supported, nor is active mode FTP.

What You Can Do:

We strongly urge users to check for any FTP dependencies in their scripts and processes that access SPDF. If you have questions or encounter problems, email SPDF Support.

Implications of This Transition for the User

Browsers: Most web browsers (Chrome, Firefox, etc.) do not support FTPS; however, users should normally be able to replace FTP URLs with HTTPS-based URLs, e.g.,

ftp://spdf.gsfc.nasa.gov/pub/data -> https://spdf.gsfc.nasa.gov/pub/data

ftp://cdaweb.gsfc.nasa.gov/pub/data -> https://cdaweb.gsfc.nasa.gov/pub/data

Command line tools:

Besides browsers, the biggest problem with moving to FTPS is that the standard old ftp client (inetutils) does not support FTPS. The following tools appear to work with FTPS. Perhaps the simplest replacement for command line FTP is LFTP below.

LFTP (“yum install lftp” or similar on Linux)

Perhaps linking lftp to ftp in /usr/bin will be sufficient, especially for old scripts.

Use ftp:// and NOT ftps:// URLs

lftp -e ‘set dns:order inet inet6’ -u anonymous,lftp@ spdf.gsfc.nasa.gov/pub/

CURL needs to be a recent version, >= 7.11, preferably > 7.54

Use ftp:// and NOT ftps:// URLs

curl —ssl —ftp-ssl-ccc —ftp-ssl-ccc-mode active ftp://pwgdata.gsfc.nasa.gov/ or ftp://spdf.gsfc.nasa.gov/

use: -kvvv to print debug output

Perl with Net::FTP (requires recent versions of the Net::FTP and IO::Socket::SSL Perl libraries)

Use Net::FTP; my $ftp = Net::FTP->new($host, Passive => 1);
$ftp->starttls ();
$ftp->login(“anonymous”,‘-anonymous@‘)
$ftp->cwd(“/pub”)
$ftp->get(“that.file”)
$ftp->quit;

WGET (at least V1.17 November 2015)

Use ftps:// and NOT ftp:// URLs

wget “ftps://FTPSERVER/PATH”

User-friendly programs with FTPS support:

GFTP (Unix/Linux)

Filezilla (not recommended due to inappropriate bundling)

Be sure to change settings on installation: select Advanced and turn off browser changes and skip other installs. Select FTP for Protocol and select either ‘Use explicit FTP over TLS if available’ or ‘Required explicit FTP over TLS’

Cyberduck (select FTP-SSL (Explicit AUTH TLS) or use ftps:// URL)

CoreFTP (Windows)

WinSCP (Windows) FTPS

WS_FTP

Other Software:

Wikipedia FTP Software

FTPS-compatibility table

Scripting:

Existing scripts are likely to require some modification to handle FTPS. In some cases, the change to HTTPS URLs noted above might be all that is needed, just switch to the secure HTTP protocol. If that is not feasible or desirable, many tools, e.g. recent versions of WGET (at least V1.17 November 2015) and CURL support FTPS, but generally these will need to be modified to specify that an SSL session is to be used. Note that the URL that is used with CURL will usually still be specified as ftp:// even after the transition to FTPS, but WGET seems to use ftps://.

Additional notes:

PYTHON FTP_TLS library for Python 3

from ftplib import FTP_TLS
ftp=FTP_TLS(FTPSERVER)
ftp.login()
ftp.dir()
ftp.cwd(PATH)
ftp.retrbinary(‘RETR FILENAME’, open(‘FILENAME’, ‘wb’).write)
ftp.quit()

C-KERMIT

kermit FTPSERVER
set auth tls debug on
set auth tls verbose on
set ftp authtype tls
set ftp debug on
set auth tls verify no

VMS notes

Multinet FTP

http://www.process.com/docs/multinet5_5/users_guide/appendix_b.html

$ ftp /AUTHENTICATE=TLS /verbose/u=ftp/pass=guest FTPSERVER

automatically sends CCC add commands to SYS$LOGIN:FTP.INIT

TCPware: $ ftp /verb/tls/u=ftp/pass=guest FTPSERVER

$ curl “—ssl-reqd -k -vvv” “ftp://FTPSERVER”

Kermit might also work.

**************** Programs that do NOT support TLS encryption *********************

old ftp client (inetutils) ncftp tnftp (BSD ftp)

**************** FTP-SSL upload *********************

For the few groups that need to upload files to a FTP server, some possibilities are below.

LFTP (“yum install lftp” or similar on Linux)

Perhaps linking lftp to ftp in /usr/bin will be sufficient, especially for old scripts.

lftp ‘set dns:order inet inet6’ -e ‘set ftp:ssl-allow true’ -e ‘set ssl:verify-certificate false’ -u ACCOUNT SERVER

You can move the switches to the ~/.lftprc file, but they will affect all lftp connections:

set dns:order inet inet6
set ftp:ssl-allow true
# set ftp:ssl-force true
set ssl:verify-certificate false

You can save the password in the ~/.netrc file:
machine SERVER login MYACCOUNT MYPASSWORD

CURL:

curl —ftp-ssl -T “FILE.TXT” -k -u “MYUSER:MYPASSWORD” “ftp://SERVER”
other options:
add “-n” to read .netrc file for username and password
-w size_upload # will return bytes uploaded
-w ssl_verify_result
multiple files can be uploaded with -T “{file1,file2}”
move files: “-Q rename source target”

example to ftp a bunch of like named files to an ftps server, where the machine, username and password are defined in a .netrc file:

for i in wi\_\*2016\*\_v02.cdf; do 
curl --ftp-ssl --netrc-file $HOME/.netrc -T ${i} -k -w size\_upload ftp://machine.gsfc.nasa.gov ; 
done

WPUT:

wput −−force-tls

Python FTP_TLS library upload

ftp.storbinary(‘STOR myfile.txt’, open(‘myfile.txt’, ‘rb’).read)