shortcut guide to secure managed file transfer

8/6/2019 Shortcut Guide to Secure Managed File Transfer

1/80


2/80

Help Your Files Find the Right OneTrue love awaits ... or your most important fles and data. The Ipswitch File Trans ersuite o managed fle trans er solutions delivers the management, en orcement,and visibility you need to enable person-to-person fle trans ers, and achieve betterbusiness results. And thats something we can all learn to love.
http://www.ipswitchft.com/


3/80

The Shortcut Guide to Secure, Managed File Transfer Don Jones

i

Introduction to Realtime Publishers by Don Jones, Series Editor

For several years now, Realtime has produced dozens and dozens of highquality booksthat just happen to be delivered in electronic formatat no cost to you, the reader. Wevemade this unique publishing model work through the generous support and cooperation of our sponsors, who agree to bear each books production expenses for the benefit of ourreaders.

Although weve always offered our publications to you for free, dont think for a moment that quality is anything less than our top priority. My job is to make sure that our books areas good asand in most cases better thanany printed book that would cost you $40 ormore. Our electronic publishing model offers several advantages over printed books: You

receive chapters literally as fast as our authors produce them (hence the realtime aspect of our model), and we can update chapters to reflect the latest changes in technology.

I want to point out that our books are by no means paid advertisements or white papers.Were an independent publishing company, and an important aspect of my job is to makesure that our authors are free to voice their expertise and opinions without reservation orrestriction. We maintain complete editorial control of our publications, and Im proud that weve produced so many quality books ove r the past years.

I want to extend an invitation to visit us at http://nexus.realtimepublishers.com , especiallyif youve received this publication from a fr iend or colleague. We have a wide va riety of additional books on a range of topics, and y oure sure to find something thats of interest toyouand it wont cost you a thing. We hope youll continue to come to Realtime for your

far into the future.educational needs

enjoy.Until then,

Don Jones
http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/


4/80

T he Shortcut Guide to Secure, Managed File Transfer Don Jones

ii

Introduction to Realtime Publishers .................................................................................................................

Ch apter 1: How to Tell if You Need Secure, Managed File Transfer ................................................... 1

What Is Secure, Managed File Transfer? ................................................................................................ 1

Fil e Transfer Scenarios ................................................................................................................................. Regularly Exchanging Files ......................................................................................................................

Occasional or AdHoc SystemtoSystem Transfers ........................................................................ 4

AdHoc, PersontoPerson Transfers ..................................................................................................... 5

Bu siness Needs for File Transfer ..................................................................................................................

Meeting Internal Requirements ............................................................................................................... 6

Meeting External Requirements ..............................................................................................................

High Availability .......................................................................................................................................

Communications Protocols ..................................................................................................................... 1

Programmability, Customization, and Workflow .......................................................................... 12

Integration with Existing Technology Assets .................................................................................. 13

Scheduling and Monitoring .....................................................................................................................

File Transfer Frequency and Volume .................................................................................................. 16

Content SecurityPreventing Malware ............................................................................................ 16

Cost ...........................................................................................................................................................

Coming Up ................................................................................................................................................

Ch apter 2: Common File Transfer Myths .................................................................................................... 19

Myth 1: Security Is Not Important............................................................................................................. 19

Myth 2: Homegrown Is Cheaper ................................................................................................................ 2

Myth 3: File Transfer Is Just FTP ................................................................................................................ Myth 4: Email Is Safe and Secure for File Transfer ............................................................................ 28

Myth 5: Slick Means Functional ..................................................................................................................

Myth 6: All Encryption Is Equal ..................................................................................................................

Myth 7: Security Is Just Encryption .......................................................................................................... 3


5/80


iii

NonRepudiation and Guaranteed Delivery ..................................................................................... 34

Auditing ....................................................................................................................................................

Authorization ...........................................................................................................................................

Retention ...................................................................................................................................................66 Coming Up Next .......................................................................................................................................

Chapter 3: Mapping Business Requirements to Technical CapabilitiesCreating Your FileTr an sfer Shopping List .....................................................................................................................................

Se curity .........................................................................................................................................................

Encryption .................................................................................................................................................

NonRepudiation and Delivery Tracking........................................................................................... 39

Logging .....................................................................................................................................................

Authentication and Authorization ........................................................................................................

Protecting Against Attacks ......................................................................................................................

Other Security Concerns ...........................................................................................................................

De ployment ...................................................................................................................................................

Hosted vs. OnPremises ............................................................................................................................

Needed Skills or Services ......................................................................................................................... Deployment Timeframe ............................................................................................................................

High Availability ...........................................................................................................................................

Int egration .....................................................................................................................................................

Supports Your Database? .........................................................................................................................

Works with Virtualization? ..................................................................................................................... 4

Works with Client Computers? .............................................................................................................. 4

W orkflow and Automation ...........................................................................................................................

Programming vs. GUIBased Workflow Building ........................................................................... 50

Automation ................................................................................................................................................

Support for Delegation ..............................................................................................................................


6/80


iv

Pr ogrammability ...........................................................................................................................................

APIs for Automation and Customization ........................................................................................... 51

Specific Programmability Needs ........................................................................................................... 51

Protocols ........................................................................................................................................................Ex ternal Connectivity ...................................................................................................................................

UsertoUser ..............................................................................................................................................

UsertoSystem .........................................................................................................................................

SystemtoSystem .....................................................................................................................................

Coming Up Next .......................................................................................................................................

Ch apter 4: Evaluating and Selecting a Secure, Managed File Transfer Solution ........................ 54

Conducting Your Evaluation ........................................................................................................................

St r ategic Tips ................................................................................................................................................

Beauty Is Only Skin Deep.........................................................................................................................

Buy for the Project, Plan for the Enterprise ..................................................................................... 58

When Is Software Like a Marriage? ..................................................................................................... 59

Cr iteria for Business Requirements ......................................................................................................... 60

Se curity Requirements ..............................................................................................................................

Encryption Levels .................................................................................................................................

Broad Security Capabilities ................................................................................................................ 6

AntiMalware ........................................................................................................................................

HighAvailability Requirements ............................................................................................................ 63

W orkflow Requirements ..........................................................................................................................

Ease of Customization .......................................................................................................................... Limits on Number of Tasks ................................................................................................................. 6

Canned Scripts and Macros ................................................................................................................ 6

Pr ogrammability Requirements ........................................................................................................... 67

Choice of API .......................................................................................................................................


7/80


v

Complexity of API ...............................................................................................................................

Pr otocol Requirements .............................................................................................................................

Choice of Protocols ...............................................................................................................................

Email as a Transport Mechanism ..................................................................................................... 69Op erational Requirements ...................................................................................................................... 6

Audit Logging and Reporting ............................................................................................................. 6

Monitoring ............................................................................................................................................

Other Considerations ................................................................................................................................

Thanks for Reading .......................................................................................................................................


8/80


vi

Copyright Statement

2010 Realtime Publishers, Inc. All rights reserved. This site contains materials thathave been created, developed, or commissioned by, and published with the permissionof, Realtime Publishers, Inc. (the Materials) and this site and any such Materials areprotected by international copyright and trademark laws.

THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND,EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,TITLE AND NON-INFRINGEMENT. The Materials are subject to change without noticeand do not represent a commitment on the part of Realtime Publishers, Inc or its web sitesponsors. In no event shall Realtime Publishers, Inc. or its web site sponsors be heldliable for technical or editorial errors or omissions contained in the Materials, includingwithout limitation, for any direct, indirect, incidental, special, exemplary or consequentialdamages whatsoever resulting from the use of any information contained in the Materials.

The Materials (including but not limited to the text, images, audio, and/or video) may notbe copied, reproduced, republished, uploaded, posted, transmitted, or distributed in anyway, in whole or in part, except that one copy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, you may not modifyor obscure any copyright or other proprietary notice.

The Materials may contain trademarks, services marks and logos that are the property ofthird parties. You are not permitted to use these trademarks, services marks or logoswithout prior written consent of such third parties.

Realtime Publishers and the Realtime Publishers logo are registered in the US Patent &Trademark Office. All other product or service names are the property of their respectiveowners.

If you have any questions about these terms, or if you would like information aboutlicensing materials from Realtime Publishers, please contact us via e-mail [email protected] .
mailto:[email protected]:[email protected]


9/80


10/80


2

Managed is one of those vague terms that can mean a lot of different things. In the context of managed file transfer, it usually refers to software solutions that are designed to facilitatefile transfer. Theres even a commonlyused acronym: MFT. MFT solutions provide awrapper around file transfer techniques and protocols, and that wrapper allows them todo things like schedule and automate file transfers, report on file transfer activity, measure

file transfer performance and other metrics, and so on.Secure file transfer is another overloaded term that people think they know the definitionof. Secure file transfer is often a component of managed file transfer; the secure part usually refers to a bunch of specific capabilities, including:

Encryption. This is what most people think of when they see the term secure filetransfer, and it refers to the ability to encode data in such a way that only thesending and receiving parties can view it.

Auditing. This is an aspect of security that fewer people tend to think of right away,but its an aspect thats becoming more and more important. It refers to the ability

to track every activity associated with file transfer, such as who sent a file, whenthey sent it, who received it, when it was received, and so on. Nonrepudiation. This refers to the ability of a file transfer system to ensure and

prove that a file was received by the correct recipient.

There are other elements of security that well explore throughout this book, but this short list will do for now.

Obviously, you need software in order to achieve secure, managed file transfer; softwarehas to provide the capabilities above and beyond those associated with simply streamingbytes across a network. In part, this book will be about the capabilities that youll

commonly find in MFT software solutions, so that you can do a better job of evaluating andselecting the right solution for your environment. If you dont really see yourself as a userof an MFT software solution, you may change your mind; Gartner has said that:

Numerous factors cause companies to reexamine how they manage the movement of information from system to system, partner to partner, and person to person. FTPalone is not a viable option to give organizations the insight, security, performance,and, ultimately, the risk mitigation necessary to responsibly conduct business.

In other words, if your business is moving data, you probably are a potential user of anMFT software solution, whether you realize it or not. This chapter, in fact, will be a sort of test for youif by the end, youre sure that none of these scenarios apply to you, thenyoure probably not going to be using an MFT solution, and you can stop reading this book (but wait until after Chapter 2, which is going to be really fun).

So now that we have a common vocabulary for secure, managed file transfer, where do youuse it?


11/80


3

File Transfer Scenarios Businesses typically have three scenarios in which they move data from place to place. It may seem a little redundant to talk about why you move data around, but this is actually animportant place to start: Why you move data is where well find the specific businesscapabilities that you need. For example, if your business never transfers data on a recurringbasis, then you may not need the same automation and scheduling capabilities that anotherbusiness requires. So lets look at three basic scenarios as well as a couple of minorvariations within each. Keep track of which scenarios apply to you.

Regularly Exchanging Files The first scenario I always think of is moving data on a regular schedule between externalbusiness partners. The main reason my mind goes to this scenario first is that it has, quitehonestly, been a major pain throughout much of my IT career. Years ago, I worked for adot com that was a virtual retailer. In other words, we sold stuff that we didnt actuallyhave. When we received customer orders, we transmitted those orders to the vendors that stocked those products, and the vendors drop shipped the products directly to thecustomer. We had a very strong need, then, to regularly transmit order information to ahuge variety of vendors, all of whom seemed to have different formats and protocols that Ihad to figure out.

Nowadays, my job would have been even more difficult. Because our customers invariablypaid by credit card, we would have been subject to the Payment Card Industry DataSecurity Standard (PCI DSS), which outlines some pretty specific technical requirementsfor how we would have to handle customer data such as addresses, phone numbers, and soon. Our file transfers to vendors would have to be not only automated but also secured so

ouldnt bthat the customers data w e revealed to anyone else.

I also had to worry about receiving files on a schedule, as our vendors transmitted invoicinginformation to us that way, although those didnt include any sensitive information. I not only had to have a place for them to send fileswhich would have been easy, because anFTP server would do the trickbut I had to watch for incoming files, grab them, and feedthem off to a batch process that would import the invoice information into our accountingsystem. I needed more than just a simple FTP server, in other words: I needed a workflowbased automation system, or invoices wouldnt get paid and our vendors would soon stopdealing with us.


12/80


4

But external partners arent the only instances when files are transmitted on a regular,recurring basis. In another job, I had to help coordinate the movement of data between anAS/400 and a Unixbased warehouse pick system. The AS/400 received sales informationfrom hundreds of retail store locations and had to generate restocking orders for thosestores. The restocking information had to be transmitted to the Unix warehouse system,

which used a system of digital indicators to tell warehouse workers which products neededto go into which box for shipment to each store. Being a Unix system, it wanted mainly todeal with file transfer via common FTP, but we needed a very controlled, managed processto be sure the information got from one computer to the other. The Unix system would alsotransmit exceptions back to the AS/400information on outofstock products, forexampleso that the AS/400 would know that a particular store hadnt yet received aparticular product and could be rescheduled for shipment when more product wasreceived in the warehouse. The whole twoway transfer of data via a fairly simplisticprotocol like FTP was a real nightmare in the beginning, and its one of the things that first

ions.set me looking at MFT software solut

So this is the first of three scenarios: Recurring,

automated

file

transfer.

Youll also see thisreferred to as system to system transfer because data isnt being transmitted betweenindividual people but is instead being transmitted directly from computer to computer. Inaddition, some automated processes are running on each computer to either produce thefiles being sent or to process the files being received.

Occasional or Ad Hoc System to System Transfers Another type of systemtosystem transferagain, not involving human beingsis thekind that doesnt occur on a regular basis but instead happens more adhoc, meaning that the parameters of the transfer are specified right when the transfer is made rather than inadvance. I did this a lot, too, in former jobs. The dot com, for example, would sometimes

need to add an extra set of orders for a specific vendor after our normal daily file transfer.That happened a lot during the holiday season when the order volumes were higher andthe vendors had earlier cutoff times. The retailer I worked for would also need occasionaladhoc transfers to the Unix system, as thats how we would download new warehousemaps and other data that didnt change very often.

Although these adhoc systemtosystem transfers needed the same kind of security andmanagement as the scheduled transfers, we found that we interacted with the file transfersystem in an entirely different way. Rather than an administrator like myself setting up thetransfer schedule in a backend management console, we found we needed a user interface(UI) that a less technicallyskilled user could operate. At the dot com, for example, adhoclate afternoon transfers were usually set up by someone in our ordermanagement teambecause by that point in the day most of the IT staff was gone or were busy with otherprojects. At the retailer I worked for, new warehouse maps were created by the warehousemanager, and he didnt like having to wait on the IT staff to get to the file transferhewanted to be able to update that Unix system as soon as the new warehouse map wascomplete.


13/80


5

Another wrinkle was introduced when the warehouse manager started delegating thewarehousemapping duty to one of his subordinates. He wanted them to set up the filetransfer to the Unix system. However, before the transfer actually happened, he wanted toreview and approve the new maps. So we had to somehow create a UI that wouldaccommodate that business workflow: accepting a file transfer order but holding it until

cific individual.approval was received from a speSo the second of three scenarios, ad hoc system to system file transfer, includesrequirements for different kinds of UIs. In some cases, you may also have additionalauditing or even workflow requirements.

Ad Hoc, Person to Person Transfers The last file transfer scenario involves people. Rather than transferring files from system tosystem, this scenario has people transferring files to each other. I did some consulting work for a hospital, and this was the most common type of file transfer there. Administratorswould transmit patient records between departments within the same hospital and wouldtransfer records between the hospital and external specialists like cardiologists,neurologists, and so on. Sometimes, they would do hospitaltohospital transfers of records,when two doctors at different hospitals needed to consult on a particular patient.

You probably wont be surprised to learn that a lot of those transfers, at least when Istarted working with them, were done via email. In the next chapter, Ill spend some timeexplaining why email is a horrible idea for this kind of transfer. At the time, the hospital Iworked with was just starting to implement their Health Insurance Portability andAccountability Act (HIPAA) requirements, and they had just figured out that email wasnt going to do the trick.

Adhoc, persontoperson transfers are tricky to deal with from a business perspective. Youhave to provide a way to accomplish them; otherwise, users will just use email attachments.If you restrict the size of attachments to make that option unworkable, theyll start usingsites like http://drop.io , which is even worse.

What we eventually figured out is that the hospital needed a system that could essentiallydo systemtoperson transfers of files, using full auditing, encryption, and the other funstuff that HIPAA required. That system needed a simple UI so that administrators couldeasily initiate transfers from their desktops, feeding the required file or files into thesystem and letting it take over and actually send the file to the destination. In the end, it felt to the end users like a persontoperson transfer: They went to a Web page on theirintranet, specified the files they wanted to send and the recipient, and the system took over

from that point and made sure it all happened. Ad hoc, person to person transfers are the third major file transfer scenario that businessessee. Which of these three scenarios are occurring in your business?


14/80


15/80


7

Our company could have defended itself against this by placing better controls on filetransfer, but thats not really the first issue. After all, the information could have beendivulged over the telephone or fax just as easilyexcept that both phone and fax were tightly managed, and using either would have created a trail of evidence leading back to theperson who broke company policies about handling personnel records. If file transfers had

been managed half as rigorously as phone and fax records, the offending employee couldhave been caughtand the company might have been able to deflect some of the legaldamage onto the person who was actually responsible. As it was, we couldnt proveanything.

This is a big area where managed file transfer is intended to help: By not only placing somerestrictions on who can send what, but most importantly by auditing what is sent, bywhom, at what time, and to where. That audit trail can prove invaluable both for internalforensics as well as in legal defense, if its ever needed.

Meeting External Requirements Im betting this is where youre expecting me to roll out the alphabet soup of industry andlegislative requirements that we all refer to as compliance, and I dont disappoint: HIPAA,SOX, GLB, FISMA, 21 CFR, PCI DSS, and more. The list is long and growing, but all of themhave common general themes when it comes to securing and managing the transfer of files.They usually all require something like this:

Data must be protected intransit to prevent unauthorized disclosure, which usuallymeans using encryption of some kind

The transfer of data must be logged in a tamperproof or tamperevident log so that auditors can see who transferred what, when they did so, whom they sent it to, andso on

Only authorized individuals should be able to access and transfer data In some cases, nonrepudiation is required, meaning that there must be proof that

the data was received by a particular system or individual so that the recipient ly decannot legitimate ny having received the data

But these are hardly the only external requirements that companies must deal with today.In many cases, external vendors or business partners may also set requirements for howtheir data must be handled. Go back to the dot com example I described earlier, andimagine that you work for one of the companies that we sent orders to. Those ordersincluded customer information, and we, as your customer, had some requirements andexpectations about how our customers information would be handled: We didnt want that information transmitted to anyone else without our permission, and we wantedaccountability for how that information was stored, accessed, used, and transmitted.Without assurances that our expectations would be met, we wouldnt do business with you.


16/80


8

In fact, customer expectations and assumptions are a major external requirement on datasecurity. Lets say you stay in a hotel in a different country. In many cases, youll be asked toshow your passport when you check in, and the hotel may record your passport number,name, address, and other personal information. Theres no worldwide rule on how that information must be protected, but you certainly expect that the hotel will keep your

personal information under wraps, and you assume that they wont go sharing it withanyone inappropriately. You might use your credit card to pay for the inroom Internet access, and you would expect that information to remain private as well, even though that information might be shared by the hotel, the Internet service provider (ISP), a billingcompany, and possibly other parties.

I definitely had those expectations and assumptions when I checked into a hotel in Europe,and used my credit card to pay for the inroom Internet access. Several weeks later, whenlarge, fraudulent charges started showing up on my account, I was a bit shocked. Aftersome investigation, it turned out that the hotel collected my billing information for theInternet access and transmitted those files in batches by unsecured, unmanaged FTP to the

Internet provider for archival purposes. Somewhere during that transfer process, the datawas accessed and several credit card numbers lifted and used for fraudulent charges.Although neither the hotel nor the Internet provider broke any local laws, they certainlyincurred the one penalty I could impose: Ill never do business with either of them again.

The fact is that many companies move all kinds of data from place to place, all the time. Itsso commonplace that we barely even think of it; its so easy in most situations that wedefinitely dont ever think twice. But simply moving data from place to place can beincredibly risky, and even if youre not violating internal company policies or legislativerequirements, you may still leave yourself open to customers wrath. Thats one of the bigreasons Gartner feels that MFT is such an important part of any business these days: Weneed to move files around, and we must do so in a secure, managed fashion.

High Availability Lets take a break from security for a bit because its hardly the only reason companies start looking at MFT solutions. High availability is another strong business driver for somethingbetter than simple FTP clients or email attachments; companies who rely on file transfersneed their file transfer solution to be available all the time.

For example, lets go back to my dot com example. Originally, I had set up a bunch of FTPscripts on one of our servers to send dropship orders to our vendors. One day, that serverstopped working. Nobody noticed because the server wasnt used for much elseit alsohad a bunch of archived product graphics and stuff, but nothing anyone had to get tocontinuously. In fact, it was a couple of days before we noticed it was downand onlybecause one of our dropship vendors called our sales manager to ask why wed stoppedsending orders every day. Oops. Obviously, we implemented monitoring solutions right away, but then I started thinking about it: Monitoring would tell me that there was aproblem, but in our line of business, we couldnt afford for there to be a problem in the first place. What we needed was a set of two servers to handle file transfers so that if one broke,the other could take over. We eventually set up something like Figure 1.1.


17/80


9

Figure 1.1: Highly available file transfer.

Basically, we had two independent file transfer systems, each with a configurationdatabase. The databases replicated with each other, so we only had to manage one of themand whatever we did would replicate to the other. The two coordinated, assigning jobs toeach other so that they both had a roughly even workload. If one went down, the otherwould just pick up all the file transfer work. Incoming file transfer connections werebalanced between them, so wed have to lose two servers before we lost the ability to sendand receive files. At first, we thought something like loadbalanced FTP clusters werekind of ridiculous, but our CEO assured us it let him sleep better at night. Literally 100% of our business depended on incoming and outgoing file transfers; it was ridiculous, he said,that we had four loadbalanced Web servers and only one file transfer server.

There are other reasons to create loadbalanced, highlyavailable file transfer farms. Onemight be to geographically distribute load. For example, if you have an office in the US andone in China, and frequently do file transfers within each continent, you might want to set up a server in the US and one in China to handle transfers within those continents. If theservers could be combined in some fashion, they could also offer failover for each other:File transfers from the US server to the Asian continent might not be as efficient, but it would be better than nothing.


18/80


10

Justification!

Its worth spending a little time thinking about what downtime in filetransfer capabilities actually cost you. For my dot com company, it literallycost us tens of thousands of dollars in refunded orders to annoyed customerswhose orders were delayed by 2 days. Knowing the cost of downtime will

make it easier for you to balance the cost of adding high availability to yourfile transfer infrastructure; there are often many ways in which an MFTsolution can be built for high availability, and knowing your cost thresholdwill help drive the necessary design decisions.

Communications Protocols Businesses file transfer needs are also strongly driven by the communication protocolsthey use to transfer files. In the ancient past of ITsay, 6 or 7 years agoit was acceptablefor companies to adopt proprietary protocols, forcing business partners to adapt to them.Today, with the wide availability of robust, open protocols, asking a business partner toswitch to a different protocol is basically a slap in the face. The problem is that there are so many open, common protocols! Once a given business adopts one, they hate switching tosomething else, so in some cases, you have to be the flexible one, offering support for as

ents.many protocols as you practically can while meeting your other business requirem

Today, ilyavailable file transfethe number of open, read r protocols is pretty large:

AS1, AS2, and AS3. These applicability statements describe how to transport datasecurely and reliably. Security is usually based upon digital certificates andencryption. AS1 is based on the SMTP (mail transfer) and S/MIME (secure fileencoding) protocols. AS2 is built around HTTP and S/MIME. AS3 utilizes FTP.

Network file copy. This is simply an automated version of dragging files from anetwork drive to a local drive or another network drive, suitable for use within anintranet environment or over a Virtual Private Network (VPN). Common protocolsfor network file copy include Server Message Blocks (SMB, used by Windows) andNetwork File System (NFS, common on Unixbased systems).

HTTP. The standard protocol for transferring Web pages between servers andbrowsers, HTTP is suitable for transmitting any kind of data. Web Services protocols(REST, SOAP, and so forth) utilize HTTP to transmit data, for example. HTTP is not intrinsically secured.

HTTPS (HTTP over TLS). By adding Transport Layer Security (TLS) to a normalHTTP connection, you can add both encryption and authentication, helping tosecure the entire connection and the data being transmitted.

FTP. The granddaddy of Internetbased file transfer, FTP is generally quick andefficient, but it lacks any kind of intrinsic security, including encryption. It isextremely widely available, however, with FTP clients installed on virtually everykind of modern computer operating system (OS).


19/80


11

FTPS (FTP over TLS *). This is an extension to FTP, adding TLS (often but incorrectlyreferred to as SSL). This is one of the most common forms of secure FTP, a termthat in practice can refer to several protocols.

Secure File Transfer Protocol (SFTP). This is a part of the Secure Shell (SSH)protocol; it is also referred to as the SSH File Transfer Protocol. This isnt quite thesame as running a normal FTP session within an SSH session (thats next), but israther a completely unique protocol. SFTP is not to be confused with the Simple FileTransfer Protocol, which is also sometimes referred to as SFTP.

FTP over SSH. Yet another secure FTP variant, this protocol tunnels a normal FTPsession through a Secure Shell (SSH) connection. This is also referred to as SecureFTP. The actual FTP traffic is unsecured, but it runs through a secured, encryptedSSH session.

Secure Copy Protocol over SSH (SCP over SSH). This works similarly to FTP overSSH, running a Secure Copy (SCP) session tunneled through an SSH connection. SCPnormally encrypts transferred data, but the SSH tunnel also encrypts authenticationand other traffic.

SMTP/POP3. Email can be used to transmit dataafter all, an email is really just data of some kind in a text format. Some organizations may send or receive data viaemail protocols, and SMTP and POP3 provide that capability.

These are the most popular and commonlyused open, Internetbased protocols in usetoday for file transfers. In fact, there are about a halfdozen other FTP variants (as if therewerent enough already), semiproprietary file transfer protocols, and more. The list Iveprovided, however, contains the protocols that 99% of companies will be using for 99% of their file transfers.

To be frank, I consider almost all of these to be musthave protocols for a file transferinfrastructure. In any company Ive ever worked for, either as an employee or a consultant,weve eventually needed to use almost all of these at one time or another. Even if we startedout only needing, say, FTPS, we would run into other business partners who preferredSFTP, or a system that could only accept data via HTTPS, or a business unit that wasreceiving information via SMTP and POP3. Its become easier for me to simply specify all of these common protocols as base requirements, as that usually leads me to a solution that will last me longer, and serve in a larger variety of business situations.

* Secure Sockets Layer (SSL) is an older protocol that has been almost universally supplanted by the newerTLS. Both protocols work similarly at a high level, and referring to TLS as SSL is almost a habit with manytechnology professionals. In practice, the two terms are used interchangeably, although it is almost alwaysTLS doing the work.


20/80


12

Programmability, Customization, and Workflow I worked for a bookstore chain at one time. This was before Amazon.com really owned thebook universe, and so we had plenty of brickandmortar competition. We didnt reallycompete on product; we all carried basically the same books, and could special orderanything else a customer might want. We didnt even compete on price; publishers set the

suggested prices for books and drive most of the promotions, so we all tended to have thesame prices and discounts on the same books. What we competed on were our business processes. We worked hard to be better at stocking our stores with new titles, restockingold ones quickly, and so on. My point is that no two companies are identical, even if theyreselling identical products at identical prices.

Because file transfer is so closely tied to business processes, you should therefore expect everyones file transfer needs to be slightly different. That means a file transferinfrastructure has to work the way your company needs it tonot in some generic fashionthat your company has to adapt to.

There are many ways in which MFT solutions accommodate different business processes.Some solutions offer an Application Programming Interface (API) that allows your ownsoftware developers to create custom file transfer consoles and clients or to incorporatefile transfers into lineofbusiness applications and other custom business processes. TheseAPIs may work for Microsofts .NET Framework, Microsofts Component Object Model,Suns Java, or some other development platform. Some vendors may offer a variety of APIsto accommodate a variety of development languages.

Vendors might provide a custom scripting language, or support existing scriptinglanguages, so that you can program your MFT solution by writing simpler scripts andbatch files. Others might provide commandline utilities that can be scripted by anexperienced administrator or programmer to customize specific operations to meet thecompanys business processes.

Another way in which MFT solutions can be customized is through workflow. Typically, thisinvolves much less expertise and overhead than programming. In addition, this optionoffers a solution to situations in which one person may queue up a file for transfer but another has to review and approve it, and you want to track all that review/approvalactivity in a log of some kind. Workflows within your business may be simplereview/approve workflows or they may be complex processes that mirror specificbusiness processes that have been defined within your organization. Solutions do varyconsiderably in how they allow you to define these workflows: Some may actually requiresome level of scripting or programming, while others may use graphical UIs (GUIs) to let you visually connect workflow components into a complete process. Im definitely a biggerfan of the graphical style of workflow construction, as it allows less technicallyskilleduserssuch as business process owners, rather than programmersto construct workflows and even maintain and modify them on an ongoing basis.


21/80


13

Integration with Existing Technology Assets Your business likely already has a significant technology investment, and adding a formalfile transfer infrastructure shouldnt require you to rebuild much of that infrastructure.Ideally, an MFT solution should integrate well with other technology assets, such asexisting Web sites for data transfer or directory services for user authentication.

There are a number of potential integration points that you can consider for your filetransfer infrastructure:

Directory services. Whether its Active Directory (AD) or some other directory, youmight want to have your file transfer infrastructure authenticate users from anexisting directory services. More and more organizations are seeking to reduce theirtotal cost of identity and access management (IAM), and solutions that utilize anexisting directoryrather than adding another user database to the environmenthelp support that cost reduction.

Databases. A file transfer infrastructure requires a database to store configuration,

job, schedule, and logging information; the ability to use existing databaseresourcessuch as an existing Microsoft, Oracle, or IBM database servercan be abenefit to some companies. Using an existing database means the file transfersolution will add less administrative overhead to the environment. However, someMFT solutions are entirely selfcontained, using their own internal database.Provided that internal database doesnt require excessive maintenance andadministration, it may not add enough overhead to worry about.

Web servers. Some MFT solutions offer Webbased UIs, especially for usersauthorized to create adhoc transfers. Figure 1.2 shows an example of a Webbasedinterface. Some solutions may be able to expose this interface through existing Web

servers; others may have an embedded Web server that requires no additionalmaintenance. Less desirable are solutions that require you to add a specific newWeb server that you otherwise wouldnt need, such as adding an Apache server toan allIIS shop or viceversa.


22/80


14

Figure 1.2: Web based file transfer UI.

Other integration points might relate to specific lineofbusiness applications, such aspredefined workflows that integrate with inventory systems, customer management systems, and so on.

Scheduling and Monitoring Scheduling is important for recurring file transfers, of course, but it can also be important for adhoc transfers, either from system to system or person to person. In other words, just because I want to set up a onetime file transfer doesnt mean I want it to happen right now; I may need it to happen later in the day or even on a specific day in the future. Yourscheduling needs may be fairly simplistic or quite complex. Figure 1.3 shows what a file

transfer infrastructure might offer in terms of a fairly simple, straightforward UI forscheduling onetime or recurring transfers.


23/80


15

Figure 1.3: UI for scheduling transfers.

Monitoring really falls into two broad categories. The first is the ability of individual usersto monitor their file transfer jobs; Figure 1.2 showed how a simple useroriented interfacemight do that. Other systems might offer users the option of receiving status emails whentheir jobs complete, run into a problem, and so on.

The other category of monitoring is the broader, IT operationslevel monitoring. IT needsto be able to monitor the health and performance of the file transfer infrastructure, receivealerts when something goes wrong, and so on. This monitoring might be accomplished by aconsole or utility specific to the MFT software that you implement. In other cases, asolution might provide monitoring that integrates with other monitoring consoles, such asHP OpenView, IBM Tivoli, or even Microsofts System Center Operations Manager. In stillother cases, the solution might simply expose monitoring instrumentation, such asWindows performance counters, which can in turn be accessed by a wide variety of monitoring tools.


24/80


16

Both types of monitoring are useful and desirable. Users will be more likely to use a filetransfer solution if they dont feel that theyre dumping their transfer requests into a big,black box; being able to check the status of their jobs and receive notifications adds a levelof confidence, and confidence leads to usage. Operationslevel monitoring is obviouslycrucial for any missioncritical service; IT needs to be able to spot upcoming problems

based on patterns and trends, and needs to be alerted quickly if something fails or goeswrong.

File Transfer Frequency and Volume This is a tricky business requirement that I think a lot of people overlookI certainly didwhen I started dealing with automated file transfers back in the day. In fact, theres a goodstory here. I used to work for a network engineering and management company, which wasa subdivision of a regional utility company. To make a long story short, we offered ourcustomers the ability to have our services appear on their utility billmuch as you might pay for satellite television on your telephone bill today. Actually making that happen was alot harder than you might think: We not only had to get our data into the right form but we

had to transmit it in a very specific fashion, at very specific times. We didnt realize howspecific those times were until, one month, we transmitted everything a couple of daysearly to work around a holiday.

It turns out that the reason our delivery schedule was so specific was that the utilitys filetransfer server was singlethreadedit could only handle one file transfer job at a time.Crazy, right? And the day we decided to send our billing information happened to be theday that another subdivision was assignedand so we basically crashed the whole system.Oops.

The moral of the story is this: Think about the frequency and volume of your file transfers,and build your file transfer infrastructure appropriately. Find out if theres a limit on thenumber of simultaneous tasks, for example, especially if youll be implementing complexworkflows for file transfer. Get a feel for the maximum sustained data throughput yourproposed infrastructure can support, and decide whether itll be sufficient for yourpurposes. If youll be handling a truly enormous amount of file data, you may even need toconsider load balancing within the file transfer infrastructure. That load balancing can alsoprovide a degree of high availability, helping meet those two business needs.

Content SecurityPreventing Malware There is a ridiculous amount of malware out there today, and a file transfer infrastructureoffers a huge opportunity for more of it to enter your environmentand an opportunity formalware in your environment to spread to other environments. Fortunately, most of what afile transfer infrastructure handles are simple data filesCSV files, XML files, and so onthat dont contain any executable code. But some files that move through yourinfrastructure will contain executable code, and you need to deal with it.


25/80


17

My preference is to not have the file transfer infrastructure implement its own antimalware measures. I have more than enough antimalware software in my environment that needs to be kept updated; I hardly need another. What I prefer to see is a file transferinfrastructure that can use the anti malware stuff I already have. In some cases, that maymean simply dumping files to disk where my antimalware software can scan it like it does

every other new file. Higherlevels of integration may allow file transfer infrastructurecomponents to actually submit incoming files to the antimalware engine before writing it to disk or doing anything else with it.

Another scenario is a file transfer infrastructure that doesnt integrate with the specificantimalware solution you have but does integrate with an existing, thirdparty, wellknown antimalware solution. Im okay with that, too, in large part because it bringsanother malwarescanning engine and technique into the environment, meaning my totalantimalware effort is more likely to catch everything.

Cost This last business driver is hardly ever the least important: How much will it cost? All of theother business concerns areand should beweighed against the cost. If high availabilitycosts significantly more than what would be atrisk for not having high availability, thenyou dont get high availability.

Ideally, a file transfer infrastructure should be somewhat modular. Businesses shouldnt beforced to buy a one size fits all solution, because no business is exactly like another.Modular componentsmaking high availability an option, for examplehelp businesscustomize a solution that fits their needs and risk mitigation requirements in a costeffective fashion. When Im building any kind of infrastructure service, including filetransfer, I like to look for solutions that offer just one or two featurerelated editions, andthen lots of options. That way, I can build what I need now, and add options later as Ineed them and can afford them. Its a bit like buying a fullsized computer over a laptop:With the laptop, you get everything in one package, but you have to be careful to buy thebest one youll ever need because theyre relatively difficult to upgrade. A fullsizedcomputer, however, can usually be opened up, and its components can be upgraded,swapped out, and so forth, all with relative ease. That means you can buy a relatively low

re.powered computer to begin with, then upgrade specific options as needed in the futu

Perhaps most important, though, is something Ill address in more detail in the next chapter: The cost of doing nothing. In other words, if you have file transfer needs, you cant just consider the cost of a file transfer infrastructure as overhead. Why?

If the business truly needs something, that something will get accomplished somehow. If it isnt through a formal file transfer infrastructure, then it will be through an informal infrastructure, often composed of cobbledtogether components, doityourself scripts, andso on. Those are not free. It can be very difficult to discern their true cost, but there isaswell see in the next chaptera cost in maintenance, risk, and so forth. Compare that withthe cost of a more formal, integrated, supported file transfer infrastructure that meets all your business needs.


26/80


18

Coming Up This chapter has been all about the reasons you might need secure, managed file transfer.Hopefully, youve seen some of your own business needs in the ones Ive outlined hereand youre at least starting to see yourself as a potential user of secure, managed filetransfer. In Chapter 3, Ill help you start mapping these business needs to the technologicalcapabilities that youll find in various file transfer solutions in the marketplace so that youcan start building an evaluation checklist. However, before I do that, I want to debunk common myths about file transferand thats whats coming up in Chapter 2. There aredefinitely plenty of myths to look at: that security isnt important, that you can build ahomegrown solution more cheaply and easily, that all kinds of securitylike encryptionare basically the same, and so forth. Well put these to the test, and figure out which

, logical scrutiny.common file transfer myths actually stand up to rigorous


27/80


19

Chapter 2: Common File Transfer Myths

As I work with consulting clients and as I speak with IT professionals at variousconferences and tradeshows, I encounter more than a few misconceptions and badassumptions related to file transfer. Some of these myths range from relatively minormisunderstandings to extremely major beliefs that actually hold back the persons entireorganization. Lets play Mythbusters and examine some of these myths. Ill look at themost common ones I run across, explain where they came frombecause many of them do,in fact, contain a nugget of truthand see how they hold up to cold, hard facts.

Myth 1: Security Is Not Important This is probably one of the first and most wildly inaccurate myths I run across. I can barelycomprehend anyone in a modern business environment believing that security really isnt

important. Todays businesses know that security is importantand therein lies the grainof truth in this myth. Todays businesses do care about security; businesses of yesteryearoften did not. In fact, even through the late 1990s, many businesses simply didnt focusvery much on security. It wasnt at all unusual for file servers to contain a permission forEveryone: Full Control at the root of their hard disks, with that permission inheriting toevery file and folder on the server. I worked for one company in the late 1990s that assigned a public IP address to every computer on the network and didnt have a firewallbetween their network and the Internet. Unthinkable a decade later, but in the past, theresimply werent quite as many security threats, and so there wasnt much security focus. In

many businesses.many respects, security wasnt important, at least not for


28/80


20

If Security Isnt Important, Then Why I enjoy having conversations with clients who start by saying, security isnt really a concern for us. In most cases, what theyre telling me is that they

dont want to use security as a blanket argument in favor of implementingsome technology solutionthey want other reasons to implement something. Which is fine. But prior to having that conversation, I usually hadto check in with a security receptionist, sign into a log, be issued a guest badge, get escorted through card keyprotected doors into a conferenceroom, and had my route monitored by security cameras. And security isnt aconcern for them?

I think security gets pulled out as a business driver so often that businesspeopleespecially business technology peoplejust get sick of it. It seemslike every IT vendor in the world tries to use security as a way of gettingtheir foot in the door or closing the sales pitch. And that can definitely befrustrating, but its disingenuous to say that security isn concern for us.t a

Trite as it is, security has to be a concern. Its rarely the only concern, but itsalways going to be there. Ive never met a single company who could happilylive without any security concerns at alleveryone locks the doors to theoffice, keeps the cash in a safe, and so on. IT security is no differentwe allget tired of hearing about it and reading about it, but its something we haveto pay attention to.

Today, of course, the world is different. There are a few more security threats out there,and weve become aware of many more security concerns than we were in the past.Nobody in their right mind would operate a network without one or more firewalls,without antimalware tools in place, and so on. There are really two reasons that todayscompanies focus a bit more on security than they did in the past: internal concerns andexternal requirements.

External requirements are fairly new, coming into play in the mid to late1990s. These areoften imposed by governments or by industry groups, typically focused on a singlebusiness industry or class of companies and are often designed to bolster consumerprotect irements include:ions or government oversight. In the US, common external requ

The Payment Card Industry (PCI) Data Security Standard (DSS) Accountability Act (HIPAA)The Health Insurance Portability and LBA)The GrammLeachBliley Act (G The SarbanesOxley Act (SOX)

agencies and contractors Various US federal requirements for government


29/80


30/80


22

So where does all of this fit in with file transfer? Security in the IT world nearly alwaysrefers to the security of information the I in IT. Information tends to leave a companyin one of two major ways: via email and via file transfer. Simply put, if youre going to find aviolation of internal or external security requirements, youve got even odds that filetransfer will be behind the violation. From a business perspective, you should really have a

couple of major security goals: Ensure that only authorized users can transfer files and that only authorized

recipients receive them. As this is often impractical, you at least want to keep track of who sends what to whom so that you can take corrective actions if necessary.

Ensure that only the intended recipient can access transferred files. In other words,you dont want unintended people snooping on your information transfers.

There are some other more subtle security goals, but Ill save those for later in this chapter.At a basic level, making sure only authorized entities have access to your information is theprimary goal of securityand it plays a strong role in file transfer.

Myth Busted

Security is important, and todays companies care. File transfer is a majoropportunity for security problems, and security has to be a majorconsideration in any file transfer scenario. In fact, security is so important,that you shouldnt even be talking to vendors who cant lead off theconversation with a rocksolid security story.

Myth 2: Homegrown Is Cheaper Most companies have reliedat least brieflyon homegrown file transfer solutionssimilar to the following example:

set locus local Avoid K95 LOCUS popup ;

ftp rawhide.redhat.com /noinit anonymous /

if fail end 1 Connection failed

i not \v(ftp_loggedin) end 1 Login failed f

ftp cd /pub/redhat/linux/rawhide/i386/Red at/RPMS/ H

i fail end 1 CD to RPMS directory failed f

set ftp dates on ; Preserve file dates

set xfer display brief ; FTP like transfer display

ftp type binary ; Force binary mode


31/80


23

set incomplete discard ; Discard incompletely received files

set ftp collision discard ; Don't download files I already have

i >= \v(version) 800205 set take error on f

; Get the files...

mget libstdc++ * glibc devel* ncurses *

mget /except:{* devel*} 4* GConf2* Glide3* LPR* MAKE* O* PyXML * SysV* V*

mget /except:{{* devel*} {a[bm]*}{anaconda*}{asp ll [a z]*}} a* e

mget /except:{{* devel*} {balsa*}{bash doc*}} b*

mget /except:{{* devel*} {chromium*}{compat [dgl]*}{cvs*}{cWnn*}} c*

mget /except:{{* devel*} {db4 [uj]*}{ddd *}{d[bd]skk*}{ esktop *}{dia *} d

{docbook style*}{doxygen*}} d*

mget /except:{{* devel*} {eel*}{emacs [el]*}{emacsp*}{epic* {evolu*}{exmh*}} e* }

mget /except:{{* devel*} {festival*}{fonts *}{freeciv*}} f*

mget /except:{{* devel*} {g[ailnt]*}{gcc [gjo]*}{gd[bm] *}{gedit*}

{gphoto2*}{gsl*}} g*

mget /except:{{* devel*} {gaim*}{galeon*}{gated*}{gtk engines*}{gtkhtml *}

{gimp print c*}{gimp [d,0 9]*}} ga* gi gt* *

mget /except:{{* devel*} {*.i686.rpm}{glade*}{glibc [dp]*}} gl*

mget /except:{{* devel*} {gnomem*}{* game*}{* user*}{* pilot*}{* audio*}

{gnucash*}{gnumeric*}} gn*

mget /except:{{* devel*} {ht[td]*}{im *}{inn *}{imap*}{isdn*}{itcl }} h* i* j* *

mget /except:{{* devel*} {k[emnopsvW]*}{kde*}{klettres*}{kt uch*} o

{kakasi *}{kappa*}{krb5 serv*}} k*

mget /except:{{* devel*} {kde i18n*}{kde[2gmtv]*}{kdeartwork*}

{kdebindings*}{kdepim*}{kdesdk*}} kde*


32/80


24

mget /except:{{* devel*} {kernel [bBdsu]*}} ke*.i386.rpm

mget /except:{{* devel*} {kmail*}{knm*}{knode*}{koffic *}{kooka*}{kppp*} e

{kstar*}} km* kn* ko* lp* ks*

mget /except:{{* devel*} {libgcj*}{libgnat*}{libtab*}{lic*}{la[mp]*}{lf p*}} l* t

mget /except:{{* devel*} {m[cguxy]*}{mailman*}{man pag s [a z]*}{mew*} e

{miniChin*}{mod_*}{mrtg*}} m*

mget /except:{{* devel*} {nautilus*} ncpfs*}{nmh *}{noatun*}{nss_*}{nut *} {

{nvi *}} n*

mget /except:{{* devel*} {*.i686.rpm}{octave*}{open[ho]*}

{openldap [cs]*}{openmotif2*} openssl0*}} o* {

mget /except:{{* devel*} {p[hvwx]*}{pan *}{perl PDL*}

{post*}{pydict*}{python d*}} p*

mget /except:{{* devel*} {qt design*}{qt2*}{quanta*}{recode*}{ruby*}} q* r*

mget /except:{{* devel*} {s[eqwy]*}{sane*}{skkd*}{snavig*}{splint*}{stard*}} s*

mget /except:{{* devel*} {sendmail doc*}{sylph*}{swig*}} se* sw* sy*

mget /except:{{* devel*} {t[oW]*}{t*fonts*}{tclx*}{tetex*}{timidity*}

{tripwire*}{tuxracer*}} t*

mget /except:{{* devel*} {unixODBC*}{uucp*}{vim [eX]*}{vnc*}} u* v*

mget /except:{{* devel*} {w[3l]*}{wine*}{wordtrans*}} w*

mget /except:{{* devel*} {x[aef]*}{xc[dhi]*}{xine *}{xmms*}{xpdf [a z]*}

{xsane*}{xtrace*}} x*

mget /except:{{* devel*} {XFree86 [cdIX]*}} X*

mget /except:{{* devel*} {zebra*}{zsh*}} y* z*

end 0

This script automates a commandline File Transfer Protocol (FTP) client. In this case, itsretrieving OS updates from a vendors servers. It works. It isnt terribly pretty, but it works.Scripts like this have existed since the advent of FTP, and they will probably continue toexist for years and years. Systems administrators rely on them to automate complextasksclearly, the task automated by this script is fairly complex, and you certainlywouldnt want to type all those commands manually on a regular basis. And this is what FTP scripts commonly look like: A series of complex commands that execute in a sequence.


33/80


25

But is a homegrown solution like this one cheaper than a commercial file transferapplication? It depends on what you mean by cheaper. A homegrown solution is certainlycheaper than running the same commands manually; the time saved by the administratoris probably pretty easy to figure out. Theres no question that automation saves time.

But a script like this can be tricky. FTP itself doesnt support a great deal of diagnosticlogging; if a single command fails, you wont necessarily know about it. If your script isimplementing a missioncritical business functionsuch as transferring data to a businesspartner on a regular basisyou may not know a problem exists until someone calls andasks where there data is.

Scripts like this can also be expensive to maintain. Its great to have someone on your teamwho has the knowledge and skills to write such a script. If theyre the only one on the team,youd better keep them happy, because if they leave, all of a sudden your convenient scriptsbecome a huge liability. If one stops working, you may not be able to fix it quicklymeaning your business may suffer.

Homegrown = Very Expensive Homegrown solutions arent limited to FTP scripts. Sometimes companiesaccidentally put themselves into the software development business to solvea particular lineofbusiness problem in their own specific way. HewlettPackard did so, using an inhouse system named Omega to keep track of commissions for the companys salespeople. Omega was the perfect exampleof a homegrown solution: It was completely outside the companys coreexpertise, was developed more than a decade ago (it started at DEC), andprobably did a great job in the beginning and was never something thecompany intended to sell. But in 2009, the company was hit with a classaction lawsuit because the homegrown solution had improperlycompensated some 2000 salespeople. The problem was simply that HP wasbigger than Omega could handle, despite efforts to keep the softwareupdated.

Christopher Cabrera of Xactly Systems says, Thats one of the big problemswith homegrown systems: they can cost you big time, often just when youneed them the most. In terms of hard dollars, homegrown systems are priceyto build, costly to maintain, difficult to economically scale, and expensive tomodify in the face of business change. In terms of opportunity costs, they lack the latest features and functionality, and they slow down and mess up vitalbusiness processes and initiatives.

Getting back to file transfer, an Information Security Magazine article quotedGartner analyst L. Frank Kenney, Most places have homegrown solutions.Its not greenfield; just about everyone has leveraged FTP. The article alsonoted that most industry experts believe, If you roll your own, the cost isvery high. And, you dont have a consistent way to manage security aroundfile transfers.


34/80


26

Often, homegrown solutions start with a pretty simple objective: move this file from here tothere. Then one day a problem occurs, and it takes a few days for anyone to notice. So thehomegrown solution is modified to include some logging and perhaps email notifications of success or failure. Then some security auditing is added. Then the script is modified tosupport encrypted connections. Server names change, so the script is modified to have the

new connection information. An OS upgrade breaks the script, so its rewritten. Over time,without realizing it, youve spent a lot of time and money on this homegrown, cheapersolution. Your company is now in the business of application development and support

.albeit in a parttime, unintended fashion

My point is this: Homegrown solutions seem cheaper at the start. They rarely stay that way,and their costs creep up on you. Homegrown tools rarely meet every business needdidyou see any security auditing in that FTP script? Any reporting? Anything to prevent use byunintended individuals? I have never been to a consulting client that didnt have some kindof homegrown file transfer tools in place, and I have never spoken with a client who didnt regret, to some degree, having such a strong dependency on those homegrown tools.

Myth Busted

If something is important to the business, then it should be done usingbusinessgrade tools, not homegrown hacks. Unless your company writes filetransfer software for a living, you shouldnt be writing file transfersoftwarelike scriptsat all.

When Homegrown Isnt Bad If youve ever looked at a commercial file transfer solution, you know manyof them support some level of automation and customizationoften through

scripts. So why arent those scripts the same as the bad ones Ive beendiscussing?

A plain FTP script starts with very little functionality beyond moving filesback and forthits not a full programming language, theres no logging, nosecurity, none of that. Adding those capabilities takes even moreprogramming, and thats where you start really spending money onsomething that you thought was free.

A file transfer solution, however, has all those capabilities built in (assumingits a decent one, of course). A script that runs within that solution is simplyinvoking prebuilt capabilities and functionality in a specific orderliterally

a script, telling the actor (the file transfer software) what lines to read (what ac to take) in a specific order. Those scripts arent as hard to maintaintionsov e because they tend to be less complex.er tim

A really good file transfer solution, however, will allow you to write scriptswithout writing scripts. Some offer graphical user interfaces and wizards tohelp build automated task sequences. Behind the scenes, the result might indeed be a script, but youll be maintaining it in a way that requires fewerspecialized skills and less effort, so youll keep your overhead lower.


35/80


27

Myth 3: File Transfer Is Just FTP One reason that many companies start with homegrown tools for file transfer is theperception that file transfer is nothing more than FTP, and that in many cases its ascheduled, automated FTP between two serverssuch as transferring data on a recurringbasis to or from a business partner. In the 21st century, though, FTP isnt the only game intown. First, there are the numerous variants of FTP that I mentioned in the previouschapter: FTPS, Secure FTP, SFTP, and so on. There are also network copy protocols, such asServer Message Blocks (SMB), Common Internet File System (CIFS), and so on. Even emailhas become a means of file transfer by simply attaching files to email messages.

You dont always get to pick which file transfer technique or protocol you use. You might behappy using FTP because its easy to automate with those homegrown scripts; a newbusiness partner, however, might insist that data be transferred using the HTTPSprotocolmeaning all your FTP scripts are useless. Another new business partner might only want to send data via email attachmentsagain negating all your FTP skills. Anotherbusiness partner might require you to transfer data according to the AS2 specification

and your onstaff FTP jockeys might not even know what that is (its a combination of HTTP and S/MIME) let alone how to crank out a script for it.

In todays business world, flexibility pays. If your business wants to transfer data to apartner, and that partner requires the use of something like AS2, which answer would you rather give your executive team?

Um, no, we dont even know what that is, let alone have the ability to write a script to do it.

Sure thing, boss.

One reason that file transfer has moved beyond FTP is that FTP is a fairly primitivetechnology. It was invented in 1971, after all, and despite numerous updates over the years,it doesnt provide a lot of the features folks needs these dayslike builtin encryption,delivery confirmation, security logging, content encoding, and so forth. FTP still has itsplace, as its fairly simple to use and is available on almost every computer OS inexistencebut it isnt the only game in town.

Myth Busted

File transfer can be a lot more than FTP, and you wont always be able toforce FTP as the file transfer solution for a given scenario.


36/80


28

Myth 4: Email Is Safe and Secure for File Transfer This is a myth Ive spent a long time arguing about. Let me first acknowledge that of coursean email can be encrypted, digitally signed, and so forth, and that most email systemssupport delivery confirmations and other feedback mechanisms. As a means of movingdata from one place to another, email is simple to use, fairly reliable, and broadlyaccessible. That does not mean it is safe and secure for businesscritical data. True,sometimes email might be your only option (if a business partner insists on it, forexample), but that doesnt mean email isnt without significant downsides.

File Transfer Isnt Just for Servers In a book like this, its very easy for me to fall into the pattern of talking about systemtoserver transfers. Thats the type of file transfer I typically work with most, as many of my clients have hired me to automate regularlyoccurring transfers, typically between one of their servers and a serverowned by one of their business partners.

Systemtosystem file transfers are, for me, easy. You plop a file transfersolution in place, set up connection parameters, pick a file transfer protocol,and set a transfer schedule. Everything happens in the background, andnobod y access to change things or mess thiny but an administrator has an gsup.

In an Information Security Magazine article, John Thielens was quoted assaying, Ad hoc file transfer is an important trend. You think of managed filetransfer as something scripted or for techies, but theres also file transfertechnology for humantohuman collaboration.

Its true. Those systemtosystem transfers are a tiny fraction of a companys

total file transfer volume. Most data is moved around by end users on an adhoc basis, most often in the form of email attachments. I cant very wellexpect end users to log into the corporate file transfer solution, set upconnection parameters, specify a transfer schedule, and so onthose endusers dont have the knowledge to do so, would probably mess up otherpeoples transfer jobs, and quite frankly dont want to go through all that hassle just to get a spreadsheet over to a business partner. Thats why emailattachments are so popular.


37/80


29

So why is email not safe and secure? Because of the w y email functions. Look at this:a

To: Don Jones m

Delivered To: [email protected]

Received: by 10.150.220.11 with SMTP id s11cs108046ybg; Mon, 28 Dec 2009 12:56:39 0800 (PST)

Received: by 10.142.2.14 with SMTP id 14mr10754165wfb.15.1262033799234; Mon, 28 Dec 2009 12:56:39 0800 (PST)

Received: from mail.wingateservices.com (mail.wingateservices.com [68.142.139.11]) by mx.google.com with ESMTP id 6si89481422pzk.103.2009.12.28.12.56.38; Mon, 28 Dec 2009 12:56:38 0800 (PST)

Received: from rsa.wingateservices.com (rsa.wingateservices.com [172.17.8.10]) by mail.wingateservices.com (8.13.1/8.13.1) with ESMTP id nBSKubrk032469 for ; Mon, 28 Dec 2009 13:56:37 0700

Those are the full message headerssomething your mail client normally hidesfrom anemail message I recently received (although Ive changed the email addresses and serveraddresses for this example). This was hardly a point to point transfer; the headers showthat my message traveled through something like four different mail servers before it got to me. Heres another:

Received: from smtpin137 bge351000 ([unknown] [10.150.68.137]) by ms214.erp.com (Sun Java(tm) System Messaging Server 7u3 12.01 64bit (built Oct 15 2009)) with ESMTP id for [email protected]; Tue, 29 Dec 2009 05:59:20 0800 (PST)

Received: from mail.abc.net ([unknown] [64.142.73.206]) by smtpin137.erp.com (Sun Java(tm) System Messaging Server 7u2 7.04 32bit (built Jul 2 2009)) with ESMTP id for [email protected] (ORCPT [email protected]); Tue, 29 Dec 2009 05:59:20 0800 (PST)

Received: from server75.appriver.com ([207.97.224.142]) by mail.abc.net with Microsoft SMTPSVC(6.0.3790.3959); Tue, 29 Dec 2009 05:59:17 0800

Received: from [10.238.8.51] (HELO inbound.appriver.com) by server75.appriver.com (CommuniGate Pro SMTP 5.2.13) with ESMTP id 954687680 for [email protected]; Tue, 29 Dec 2009 08:59:10 0500

Received: from tippit.wc09.net ([74.203.49.55] verified) by inbound.appriver.com (CommuniGate Pro SMTP 5.1.7) with ESMTP id

641729304 for [email protected]; Tue, 29 Dec 2009 08:41:39 0500 Received: from aweb06.whatcounts.com (172.16.2.16) by

tippit.wc09.net (PowerMTA(TM) v3.5r15) id h783j60kup8j for ; Tue, 29 Dec 2009 05:14:12 0800


38/80


30

This time, the message went through six servers before it got to its destination. Thats howemail worksin fact, its a major feature of the Simple Mail Transfer Protocol (SMTP) that powers Internet email. The idea is that mail can take many routes to get to its destination,so if one route is down, the mail can still get through.

Thats all well and good when your email is a message to Mom, but when youre talkingabout sensitive business information, emails roundabout paths present some distinct disadvantages:

All this bouncing from server to server takes time. How often have you been on aconference call, sent an email to someone else on the call, and then waitedpatiently, Im surewhile they hit refresh over and over on their mail client?Critical business information may need faster, pointtopoint delivery.

Email doesnt actually support true delivery confirmation. In other words, you thesender cant hand the email to the recipient and know that its in the recipientshands. What email can doif the recipients email system supports itis send

another email back to you letting you know they got it. Those confirmations canobviously be forged, mislaid, misrouted, or eliminated enroute. Direct, pointtopoint transfers provide actual delivery confirmation.

Your email messages pass through many hands, and each server on the route cankeep a permanent copy. In practice, most legitimate mail servers never do becauseof the space that would entail. But they can. And even if youve encrypted the emailcontents, giving someone a copy of the message will give them time to break that encryption. Older 40bit encryptionwhich is still in widespread usecan bebroken by a home computer in a few days. Youll never know that the decryption isin progress because a copy of your email will have been forwarded to its intendedrecipient.

Even encrypted email messages contain certain information that might beconsidered sensitive. For example, the sender and recipient identities cant beencrypted because that information is needed to deliver the message. Anyoneintercepting the messagewhether its encrypted or notwill know that thoseparties are in discussions. That knowledge helps someone involved in industrialespionage, for example, focus their efforts on the most promising messages.

For companies that spend thousands of dollars on card keys for their office doors; tens of thousands on co

shortcut guide to secure managed file transfer

Documents