usb/ip: a transparent device sharing technology over ip

Vol. 46 No. SIG 12(ACS 11) IPSJ Transactions on Advanced Computing Systems Aug. 2005

Regular Paper

USB/IP: A Transparent Device Sharing Technology

over IP Network

Takahiro Hirofuchi,† Eiji Kawai,† Kazutoshi Fujikawa†

and Hideki Sunahara†

Personal computing with affordable computers and their peripheral devices becomes morepopular. To use such devices more efficiently and improve their usability, people want toshare surrounding peripheral devices between computers without any modification of theirown computing environments. The recent device sharing technologies in the pervasive com-puting area are not sufficient for the peripheral devices of personal computers because thesetechnologies do not provide the network-transparency for applications and device drivers. Inthis paper, we propose USB/IP as a transparent device sharing mechanism over IP network.This advanced device sharing mechanism is based on the modern sophisticated peripheralinterfaces and their supports in operating systems. By the Virtual Host Controller InterfaceDriver we have implemented as a peripheral bus driver, users can share diverse devices overnetworks without any modification in existing operating systems and applications. The ex-periments show that USB/IP has fairly practical I/O performance for various USB devices,including isochronous ones. We also describe the performance optimization criteria for thefurther improvements.

1. Introduction

By the innovations in computing technology,people have their own computing environmentwith several personal computers. Users alwayswant to use surrounding peripheral devices on-demand and seamlessly through their own com-puters and their favorite applications, even ifthese devices are already attached to other com-puters. For example, a user who brings backhis mobile computer to his office may want tomake the backup of his working files directlyinto a DVD-R drive of a shared computer at theoffice, rather than directly use the shared com-puter or reconnect the DVD-R drive to his com-puter. At his desk, he may also wish to workon his mobile computer with an ergonomic key-board and a mouse which are already-attachedto a desktop computer without plugging to aKVM switch. In the context of resource man-agement, the key technology for these scenariosis the network-transparent device sharing mech-anism by which computers can interact seam-lessly with other computers’ devices as well asdirectly-attached ones.

Many device sharing technologies 10),16) havebeen proposed in the pervasive computing areato aggregate accesses to network-attached de-vices and improve their usability. These tech-nologies address dynamic discovery, on-demand

† Nara Institute of Science and Technology

selection and automatic interaction among de-vices. However, the network transparency forexisting device access interfaces, by which ex-isting applications can also access to remoteshared devices without any modification, hasnot been addressed adequately.

In this paper, we propose USB/IP as a trans-parent device sharing mechanism over IP (In-ternet Protocol) network. The main componentof this extension is the Virtual Host ControllerInterface driver implemented as a peripheralbus driver. It is built in the lowest layer in op-erating systems so that most applications anddevice drivers can access remote shared devicesthrough existing interfaces after these devicesare virtually attached to a computer. We be-lieve that this approach is fairly practical be-cause of recently emerging sophisticated periph-eral interfaces and broadband networks.

Our device sharing approach presents severaladvantages over the conventional approaches.First, a computer can access the full functional-ity of a shared device. The control granularityof the shared device is the same as directly-attached one. In our system, all the low-level

The original paper 7) was published in the Pro-ceedings of the FREENIX Track: USENIX An-nual Technical Conference (The USENIX Associ-ation: Berkeley, CA, April 2005). Some parts ofthe FREENIX paper are refined for readers’ conve-nience. Especially, Section 2, Section 3, and Sec-tion 5 of the original paper are reconstructed.

349

350 IPSJ Transactions on Advanced Computing Systems Aug. 2005

Fig. 1 USB device driver model.

I/O data for the devices are encapsulated intoIP packets and then transmitted. Second, acomputer can access a shared device with itsoperating system and applications. With only afew additional device drivers, users can controlthe shared device as if it was directly attachedto a local peripheral bus. Last, various comput-ers with different operating systems can sharetheir devices with each other, because the low-level device control protocols do not depend onany structures of operating systems.

In the remainder of this paper, we expandon our vision of USB/IP. Section 2 explainsthe overview of USB/IP and then gives the de-sign details. Section 3 shows the evaluation ofUSB/IP and clarifies its characteristics. Sec-tion 4 examines related work. We conclude ourstudy in Section 5. The availability of USB/IPis noted in Section 6. The contribution of thispaper is that we show the key idea of USB/IP isenough practical by implementing USB/IP onLinux and evaluating its I/O performance.

2. USB/IP

2.1 USB Driver ArchitectureUSB (Universal Serial Bus) 17) is one of the

sophisticated peripheral interfaces based on therecent hardware progress. In the USB 2.0specification announced in April 2000, a hostcomputer controls various USB devices with3 transfer speeds (1.5Mbps, 12.0 Mbps, and480Mbps) and 4 transfer types (Control, Bulk,Interrupt, and Isochronous). These transfersare serialized and controlled by a dedicatedhardware function which is named USB HostController. Figure 1 shows the USB devicedriver model in most operating systems. A USBHost Controller Driver (USB HCD) exists inthe lowest layer of the device driver stack andabstracts the I/O interface of a host controller

into a common API for USB device drivers. AUSB PerDevice Drivers (USB PDD) is respon-sible for the control of each USB device. Thekey design of the USB driver stack is that aUSB Host Controller and its USB HCD provideUSB PDDs with abstracted data input/outputfor I/O buffers. Although a USB device andits USB PDD use the buffer to transfer somecontrol data and corresponding reply data, theUSB HCD do not distinguish between controland reply.

In USB device drivers, a USB Request Block(URB) presents USB I/O in a controller-independent form, which includes informationabout I/O;• I/O buffer• I/O direction (Input/Output)• I/O speed (1.5 Mbps/12 Mbps/480 Mbps)• I/O type (Control/Bulk/

Interrupt/Isochronous)• I/O destination address• completion handler

An application or a device driver controls aUSB device as follows:( 1 ) A USB PDD converts I/O requests from

another driver into URBs and submitsthese URBs to a USB HCD.

( 2 ) The USB HCD transfers data as de-scribed by the URB.

( 3 ) After I/O of the URB is completed, thecompletion handler of the URB is calledin an interrupt context.

( 4 ) The USB PDD notifies the upper driverof the requested I/O completion.

A USB device and its USB PDD use 4 differ-ent transfer types depending on characteristicsof the device. Control and Bulk transfer typesare asynchronously scheduled into the rest ofthe bandwidth after the periodical transfers.Control transfer is the most fundamental onefor enumeration and initialization of devices.In 480 Mbps mode, 20% of the bandwidth isreserved for Control transfer. Bulk transfer isused for the requests without any temporal re-striction, such as storage device I/O, which isthe fastest transfer when the bus is available.Isochronous and Interrupt transfer types are pe-riodically scheduled. Isochronous transfer canmove control data at a constant bit rate, whichis useful to read image data from a USB cam-era or to write sound data to a USB speaker.Interrupt transfer confirms the maximum delayof the requested I/O. This is used for USB miceand keyboards, which move a small amount of

Vol. 46 No. SIG 12(ACS 11) USB/IP 351

data sporadically.2.2 Overview of USB/IPTo share various USB devices between com-

puters, we propose the Virtual Host ControllerInterface driver as one of the USB HCDs. TheVHCI driver provides the same access interfaceto a virtually-attached remote USB device. Itencapsulates USB requests into IP packets andtransmits them to the remote device. This ap-proach makes the best use of attractive USBfeatures, such as various device supports anddynamic device configuration. Furthermore, inthe context of device sharing architecture, theadvantages of USB/IP are summarized as fol-lows:

Full Functionality. All the functions of re-mote devices can be manipulated by the oper-ating system. The abstraction on the layer ofUSB HCDs conceals only the difference betweenUSB host controllers. Per-device operations arefully transferred to remote USB devices.

Network Transparency. The VHCI driverconceals the implementation details of networksharing mechanisms. Other device drivers (e.g.,file system, block I/O and virtual memory) andapplications do not recognize any difference ofthe access interfaces between remote devicesand locally-attached ones.

Interoperability. If the VHCI driver is im-plemented in other operating systems, hetero-geneous computers can share their USB de-vices. The USB protocols, which are definedby the several standards, do not depend on thestructures in an operating system. Addition-ally, most operating systems have quite similarUSB driver models. Basically, USB/IP is fairlyapplicable to various operating systems.

To transfer USB I/O data over IP networkefficiently, in which transfer delay and jitter arelarger than the USB wire, the key point is thatUSB/IP encapsulates URBs into IP packets. Ingeneral, the fine-grained I/O operations withsmall data size and short temporal restrictiondepress the total I/O performance when eachoperation spends more execution time. The na-tive I/O granularity of USB is too small to con-trol USB devices over IP network. Isochronoustransfer needs to move 3KB data in every mi-croframe (125 us) constantly and Bulk transferneeds to move 6.5KB data in a microframe.The I/O in a microframe is named as “trans-action”, which is given in Transaction Descrip-tor (TD). However, a URB, which substitutesa series of USB I/O transactions with one re-

Fig. 2 USB/IP design.

quest by concatenating them into its I/O buffer,can relax the restrictions to be ease to handlesmall I/Os. For example, if a URB represents80 I/O transactions of Isochronous transfer andeach transaction is executed every microframe,the temporal restriction of the I/O is relaxed to10 ms, still keeping its I/O granularity 125 us. Ifa URB has the 100 KB buffer of Bulk transportand each transaction moves 512 B, this URBsubstitutes for 200 I/O transactions.

Considering the bandwidth of Gigabit Ether-net goes beyond that of 480 Mbps of USB 2.0, itis possible to apply the URB-based I/O modelto control remote USB devices over IP network.The evaluation of using URBs for USB/IP isdescribed in Section 3.

2.3 Design and ImplementationThe design of USB/IP is illustrated in Fig. 2.

We create the VHCI driver as a USB HCD ina client host and the Stub driver as a USBPDD in a server host. The VHCI driver em-ulates the USB Root Hub’s behavior; whena remote USB device is connected to a clienthost over IP network, the VHCI driver noti-fies the USB Core Driver of the port statuschange. The Stub driver is responsible forexporting of shared devices. It is automati-cally loaded when the devices are attached tothe server computer. Most URBs are encapsu-lated/decapsulated into/from IP packets whiletheir device numbers are always converted be-tween the client’s one and the server’s one.

The USB/IP implementation on Linux usesa common API of a USB core driver in bothclient and server sides. First, a USB PDDsubmits a URB by usb submit urb(struct*urb, ..). usb submit urb() calls theurb enqueue(struct *urb, ..) of the VHCIdriver after small sanity checks. Theurb enqueue() translates a URB into a SUB-


Table 1 USB/IP protocol data unit (SUBMIT and RETURN).

Byte Field0–3 SUBMIT4–7 bus number8–11 device number12–15 sequence number16–19 IN/OUT & I/O type & endpoint20–23 transfer flags24–27 buffer length28–31 number of included transaction32–35 transaction interval36–39 control buffer40– isochronous descriptors (if available)

I/O buffer (if available)

Byte Field0–3 RETURN4–7 bus number8–11 device number12–15 sequence number16–19 reserved20–23 transfer flags24–27 buffer length28–31 number of included transaction32–35 error count36–39 transfer status40– isochronous descriptors (if available)

I/O buffer (if available)

MIT PDU (Protocol Data Unit) (Table 1) andthen transmits it to a remote Stub driver. Next,the Stub driver receives the SUBMIT PDU,creates a new URB from it, and then sub-mits the URB to a real USB host controller byusb submit urb(). After the requested I/O ofthe URB is completed, the Stub driver sets upa RETURN PDU which includes the status ofI/O and input data if available, and then trans-mits it to the VHCI driver. Finally, the VHCIdriver notifies the USB PDD of the completionof the URB. It also noted that the USB driverstack and the USB/IP implementation allow aUSB PDD to enqueue multiple URBs simulta-neously for I/O performance improvement.

An existing RPC mechanism, such as SunRPC, can be used to transfer a URB. However,our prototype implementation simply transfersa URB by defining SUBMIT and RETURNPDUs. This implementation approach is quitesimple and practical enough to study the firstfeasibility of USB/IP.

All PDUs for a virtually-attached USB de-vice are transferred by a TCP/IP connection.In the prototype implementation, the TCP/IPconnection is established by userland softwareand its socket descriptor is passed to the VHCIand Stub driver via /proc file system interfaces.To transmit the TCP/IP packets as soon as pos-sible avoiding the buffering delay, the Nagle al-gorithm 13) ☆ is disabled. The current USB/IPimplementation does not use UDP/IP commu-nication. The characteristics of the transmis-sion errors of USB and UDP/IP are quite differ-ent. Though the host controller does not resub-mit failed Isochronous transactions, USB PDDsand devices expect that most transactions suc-

☆ This algorithm states that no small packets will besent on the connection until the existing outstandingdata is acknowledged.

ceed. These transaction failures seldom occurunless broken devices or cables are used. There-fore, the transport layer for URBs must guar-antee the data arrival in order and retransmitlost packets.

To use remote devices over IP network, theappropriate error recovery should be consid-ered. The error recovery of USB/IP exploitsthe semantics of USB. If the TCP/IP con-nection is suddenly disconnected, the VHCIdriver detaches the device virtually and theStub driver resets the device. In the same wayas directly-attached USB devices, some applica-tions and drivers may lose their data. However,this recovery policy is appropriate to LAN en-vironments, in which sudden disconnection ofTCP/IP seldom occurs. The USB/IP imple-mentation is kept simple by this policy.

The current USB/IP implementation doesnot allow multiple computers to access a sharedUSB device simultaneously. This design crite-rion is logical because the USB architecture isdesigned to provide one computer with periph-eral device access. At the beginning of USB/IPaccess to a shared device, a computer needsto connect the device exclusively. If the de-vice is already used by another computer, thecomputer cannot use the device until anothercomputer disconnects the device. Even thoughshared device access is exclusive, USB/IP givesa fair degree of transparency for remote shareddevices.

We have implemented USB/IP for Linux Ker-nel 2.6 series. A simple USB/IP applicationis also developed as in Fig. 3. As this figureshows, a user selects a shared USB device ona server machine and attaches it to a clientcomputer. To attach a remote shared devicevirtually, a userland program devconfig in theclient initiates a TCP/IP connection and passesan established TCP/IP socket to the VHCI


% devconfig list list available remote devices3: IO-DATA DEVICE, INC. Optical Storage: IP:PORT : 10.0.0.2:3000: local_state : CONN_DONE: remote_state : INUSE: remote_user : 10.0.0.3: remote_module: USBIP

2: Logitech M4848: IP:PORT : 10.0.0.2:3000: local_state : DISCONN_DONE: remote_state : AVAIL: remote_user : NOBODY: remote_module: USBIP

% devconfig up 3 attach a remote DVD Drive% ...% ... mount/read/umount a DVD-ROM% ... play a DVD movie% ... record data to a DVD-R media% ...% devconfig down 3 detach a remote DVD Drive

Fig. 3 USB/IP application usage.

Table 2 Machine specifications for experiments.

CPU Intel Pentium III 1GHzMemory SDRAM 512MBNICs (client/server) NetGear GA302TNICs (NIST Net) NetGear GA620USB 2.0 Interface NEC µPD720100

driver via /proc file system. In this paper, wefocus on the basic architecture of USB/IP andits performance evaluation. The device discov-ery, the authentication and the security will bedescribed in another paper.

3. Evaluation

In this section, we show the USB/IP char-acteristics. We have conducted several exper-iments to measure the USB/IP performance.The used computers are listed in Table 2. Toemulate various network conditions, we use theNIST Net 1) package on Linux Kernel 2.4.18.A client machine and a server machine areconnected via a NIST Net machine, shown inFig. 4. Both the client and server machinesrun Linux Kernel 2.6.8 with the USB/IP ker-nel modules.

3.1 Performance Evaluation of DataSink/Source

In this subsection, we used a pure sink/sourceUSB device rather than a consumer USB de-vice, to clarify the characteristics of USB/IPitself. A USB peripheral development boardwith a Cypress Semiconductor EZ-USB FX2chip 3) was programmed to be a sink/sourceof USB data. When programmed as a datasink, the board receives output data of Bulkor Isochronous transfer types from a computer.When programmed as a data source, the boardsends input data of Bulk or Isochronous trans-fer types to a computer. We wrote its firmwares

Fig. 4 Experiment environment.

and the test device driver as a USB PDD for theexperiments.

3.1.1 Bulk TransferThe I/O performance of USB/IP depends on

network delay and the data size of the opera-tion. In this subsection, we derive the modeledthroughput from the results of experiments andgive the optimization criteria.

The USB/IP overhead for USB requests ofBulk transfer was measured. As Fig. 5 shows,the test driver submits a Bulk URB to the re-mote source USB device, waits for its comple-tion and then resubmits it continuously. Eachexecution time of URB in the client was mea-sured by the TSC register of Intel Pentium pro-cessors, changing URB data size and networkRTT. When NIST Net sets network RTT to0 ms, the actual RTT between a client machineand a server machine is 0.12 ms by ping.

The result is shown in Fig. 6. The executiontimes of URBs are linear to the data size andthe gradients in each case are constant. TheCPU cost for the USB/IP encapsulation is quitelow; in these experiments, it is always under afew percentages. There is no influence of theTCP/IP buffering because TCP NODELAY is setin the socket options. From the graph, the ex-ecution times toverIP for the data size s is

toverIP = aoverIP × s + RTT .aoverIP is the gradient value in the USB/IPcases. Then the throughput thpt is

thpt =s

toverIP=

s

aoverIP × s + RTT. (1)

The regression analysis shows aoverIP is 6.30e-2 ms/KB and the intercept in the 0 ms case is0.24 ms. The modeled throughput by Eq. (1)is illustrated in Fig. 7. The actual throughputfrom the experiments is illustrated in Fig. 8.As these graphs show the same shape, the


Fig. 5 The summarized driver behavior of the experiments in Section 3.1.1.In the case of USB/IP, the enqueued URBs are transferred to theserver’s HCD between 1 and 2, and vice versa between 3 and 4.

Fig. 6 Execution time of a URB.

Fig. 7 Throughput from model.

throughput of the model is fully substantiatedby the experimental results. Within the pa-rameter range of the experiments, this modelis appropriate to estimate the throughput withURB data size and network RTT.

To summarize these experiments, we have es-timated the appropriate URB data size under

Fig. 8 Throughput from experiments.

various network delays. With the additionalexperiments in which multiple URBs were en-queued simultaneously, we have confirmed thatthe throughput of Bulk transfer is dependenton the total I/O data size of simultaneously-enqueued URBs. To keep enough throughputwith some network delay, USB PDDs shouldenlarge each URB data size or the queuingdepth of URBs. Moreover, when a large amountof URBs are enqueued asynchronously underlarger network delay, the TCP/IP window sizemust be also enlarged to fill up the networkpipe.

3.1.2 Isochronous TransferIn this subsection, we examine the isochrony

of USB/IP. To keep periodical transfers forUSB Isochronous devices, the starvation oftransaction requests must be avoided in thehost controller. Therefore, the USB drivermodel allows USB PDDs to enqueue multipleURBs simultaneously. In the case of USB/IP,


Fig. 9 The summarized driver behavior of the experiments in Section 3.1.2.In the case of USB/IP, the enqueued URBs are transferred to theserver’s HCD between 1 and 2, and vice versa between 3 and 4.

it is important to select the appropriate queu-ing depth of URB, as determined by networkdelays.

We wrote the firmwares and the test devicedriver for an Isochronous source device. Thedevice and drivers are configured as follows:• A transaction moves 512 B data in one mi-

croframe (125 us).• A URB represents 8 transactions.

In this case, the completion handler of the URBis called every 1 ms (125 us × 8) ☆. Figure 9shows the detail of the driver behavior in theexperiments. The USB PDD sets up each URBwith the pointer of I/O buffer for 8 transactionsand then enqueues multiple URBs. As a result,while the host controller keeps pending I/O re-quests in the Periodic Frame List, the comple-tion handler is called every 1 ms and the HCDmoves the input data to the USB PDD periodi-cally. On the other hand, if there is no pendingI/O request in the Periodic Frame List, the hostcontroller does not copy data and isochronousdata will be lost.

For the directly-attached source device, whenthe USB PDD submitted only one URB, thecompletion intervals became 11.1 ms because ofthe starvation of the requests. When the USBPDD submitted 2 or more URBs simultane-ously, the completion intervals kept 1 ms con-

☆ The 1 ms interval in the experiments is selected tobecome enough small to examine the isochrony ofUSB/IP. In general, the interval of completion isdefined by driver developers as approximately 10msconsidering the smoothness of I/O and the process-ing cost.

Fig. 10 Mean completion intervals.

stantly. The standard deviations were approxi-mately equal to 20 ns for any queuing depth.

In the case of USB/IP, Fig. 10 illustrates themean completion intervals for various networkRTTs and the queuing depths of submittedURBs. Even under some network delays, theUSB PDD could get 1 ms completion intervalswith the enough queuing depths. For example,when the network delay is 8 ms, the appropriatequeuing depth is 10 and more. In this condition,time series of completion intervals are figuredin Fig. 11. Immediately after the start of thetransfer, the completion intervals vary widelybecause the host controller is not still filled withenough URBs. Once the host controller is filledand the cycle becomes stable, the completionintervals keep 1 ms. Figure 12 shows the stan-dard deviations of completion intervals. Withthe sufficient URBs, the measured standard de-viations are less than 10 us, including the NISTNet’s deviations 1). These values are less than


Fig. 11 Time series data of completion intervals(RTT = 8ms, queuing depth = 10).

Fig. 12 Standard deviations of completion intervals.

one microframe interval (125 us) and enough ac-curate for most device drivers, because processscheduling is driven by jiffies, which is incre-mented every 1 ms in Linux Kernel 2.6 series.There is no influence of the TCP/IP bufferingbecause of the socket option TCP NODELAY.

The periodical transfers of USB/IP are illus-trated in Fig. 13. In this case, the USB PDD inthe client host enqueues 3 URBs which are com-pleted every xms. For periodical completionsin the server host, the host controller must al-ways keep multiple URBs enqueued. Therefore,with the queuing depth q, the time in which thenext URB is pending is

tnpending = (q − 1)x − RTT > 0.Then, the appropriate queuing depth q is cal-culated by

q >RTT

x+ 1. (2)

In Fig. 10, Eq. (2) can explain the sufficientqueuing depth of URBs in the experiments.

To summarize these experiments, USB PDDswith periodic transfers must enqueue multipleURBs to fill the queuing depth q of Eq. (2)

Fig. 13 USB/IP model for periodical transfers.

Fig. 14 usbview output for a USB/IP device.

for continuous stream I/O. In the other net-works rather than LAN, with jitter or packetloss, q should be increased with sufficient mar-gins. The result can be also applied to Inter-rupt URBs which specify the maximum delayof completion. This examination continues forcommon USB devices over IP network in Sec-tion 3.2.2.

3.2 Performance Evaluation of USBDevices

In this subsection, we show the USB/IP char-acteristics for common USB devices. All USBdevices we have tested can be used as USB/IPdevices. Figure 14 shows that a client host vir-tually attaches a remote USB camera throughthe VHCI driver and a USB device viewer


Table 3 Specification of the tested USB hard disk.

Product Name IO-DATA HDA-iU120Interface USB 2.0/1.1, iConnectCapacity 120GBRotation Speed 5,400 rpmCache Size 2 MB

usbview 11) gets the device descriptors. Nodifference between USB and USB/IP can beseen except that the host controller is VHCI.USB PDDs can also control their correspond-ing remote USB devices without any modifica-tion themselves. In our LAN environment, theperformance degradation of USB/IP is negligi-ble for actual usages. The specific details ofeach kind of USB/IP devices are described inthe next paragraphs. It should be noted thatthe performance of USB/IP becomes worse un-der a congestion network. However, this issuewill be addressed in future work. For exam-ple, to transfer USB requests beyond wide areanetwork, a reservation mechanism of networkresources may be employed with USB/IP.

3.2.1 USB Bulk DeviceUSB storage devices (e.g., hard disks, DVD-

ROM drives, memory drives, etc.), USB print-ers, USB scanners and USB Ethernet devicesuse USB Bulk transfer mainly. We can usethese devices by USB/IP; for USB storage de-vices, we can make partitions, file systems,mount/umount and operate files. Moreover, wecan play DVD videos in remote DVD drives andcan also write DVD-R media, with existing ap-plications. As described in Section 3.1.1, theUSB/IP performance of USB Bulk devices de-pends on the queuing strategy of Bulk URBs.We have tested the original USB storage driverof Linux Kernel 2.6.8 to show its effectivenessfor USB/IP.

The experiment environments are the sameas those described in Section 3.1.1. NIST Netwas used to emulate various network delays.We have run Bonnie++ 1.03 2) benchmarks forext3 file system on a USB hard disk describedin Table 3. Bonnie++ measures performancesof hard drives and file systems, by the file I/Otests and the file creation/deletion tests. Thefile I/O tests include sequential I/O per char-acter and per block and random seeks. The filecreation/deletion tests do creat/stat/unlinka lot of small files.

Figures 15 and 16 show the sequential I/Othroughput and their CPU usages on the clientby USB and USB/IP, respectively. Figure 17

Fig. 15 Bonnie++ Benchmark (sequential Read/Write throughput on USB and USB/IP).

Fig. 16 Bonnie++ Benchmark (sequential Read/Write CPU usage on USB and USB/IP).

Fig. 17 Bonnie++ Benchmark (sequential Read/Write throughput on USB/IP).

shows sequential I/O throughput by USB/IPunder various network delays. For char writeand block write, the throughput by USB/IPwhen NIST Net’s RTT is 0ms (0.12ms by ping)is approximately 77% of the throughput by onlyUSB, for rewrite 66%, for char read and blockread 79% and 54%, respectively. Because thegraph shows that there is enough room in theCPU usage, the performance degradation of theUSB/IP case can be attributed to insufficientqueuing data size. The Linux USB storagedriver is implemented as a glue driver betweenthe USB and SCSI driver stacks. For the SCSIstack, the USB storage driver is a SCSI hostdriver. A SCSI request with scatter-gather listsis repacked into several URBs which are respon-sible for each scatter-gather buffer. The LinuxUSB storage driver does not support the queu-


Fig. 18 Bonnie++ Benchmark (random seek speedon USB/IP).

Fig. 19 Bonnie++ Benchmark (Create/Delete speedon USB/IP).

ing of multiple SCSI requests. Therefore, a to-tal I/O data size of URBs submitted simulta-neously is the same as each SCSI request size;in the case of the block write, this is approxi-mately 128 KB. This queuing data size is smallfor USB/IP under some network delays, as wehave considered in Section 3.1.1. To optimizethe sequential I/O throughput for USB/IP, thereasonable solution is that the USB storagedriver should provide the SCSI request queu-ing.

The throughput of random seek I/O byUSB/IP is illustrated in Fig. 18. This test runsa total of 8000 lseek in a file randomly withthree seek processes. A seek process reads ablock in each case and also writes back the blockin 10% of cases. The throughput by only USB is167KB/s. The throughput difference betweenUSB and USB/IP is quite smaller than thatof sequential I/Os. The CPU usages in boththe USB and USB/IP cases are 0%. Randomseek I/O has its bottleneck in the seek speedof a USB hard disk itself. The seek speed ofthe hard disk is relatively slower than thoseof read/write I/Os; the rational speed of USBhard disk tested is 5,400 rpm and its seek speedis approximately 10 ms.

Figure 19 shows the speed of file creationand deletion by USB/IP under various networkdelays. The speeds by USB are 682/s in the

sequential creation, 719/s in the random cre-ation and 2,687/s in the deletion. In all thecases, the CPU usages are over 90%. The dif-ferences between USB and USB/IP are quitesmall and the speeds of each test are almostconstant under various network delays. In thefile creation/deletion tests, the bottleneck is theCPU resources.

3.2.2 USB Isochronous DeviceUSB multimedia devices, such as USB cam-

eras and USB speakers, use USB Isochronoustransfer to move their periodical data. Wehave tested a USB camera with the OmniVisonOV511 chip and a USB Audio Class speaker.These devices work completely by USB/IP inour LAN; we can capture the video from thecamera and play some music by the speaker.

The Linux USB Audio Class driver submits 2URBs simultaneously for multi-buffering. EachURB is responsible for 5ms transactions. Equa-tion (2) shows this driver can play the speakerby USB/IP under less than 5 ms network de-lays. For larger network delays, the easy andeffective solution is that the completion inter-val of each URB should be enlarged. However,there is also the drawback that such modifica-tion makes the I/O response worse than ever.

3.2.3 USB Interrupt DeviceUSB Human Input Devices, such as USB key-

boards and USB mice, use USB Interrupt trans-fer to move data sporadically like IRQ. Someother devices also use the Interrupt transfer tonotify their status changes. In our LAN, we canuse these devices by USB/IP comfortably withconsoles and X Window System.

Most USB HID drivers submit only one URBwhose completion delay is 10ms. After theURB is completed, the driver resubmits it. Thedrivers read the interrupt data every 10 ms,which is accumulated in the device endpointbuffer. Under large network delays, the pos-sible pitfall is that the device endpoint buffermay overflow. Under more than 100ms net-work delays, the users may feel odd for theirinput devices. The former problem can be re-solved by enqueuing more URBs not to overflowthe endpoint buffer. The latter problem is un-derlying in any network program with humaninteraction.

4. Related Work

iSCSI 14) transports the SCSI packets overTCP/IP and provides the access to storage de-vices beyond IP networks. It is worthy of re-


mark that iSCSI is the extension of a SCSIbus to IP networks and its protocol has basi-cally network transparency and interoperabil-ity. However, iSCSI supports only storage de-vices. Our USB/IP has the advantage that alltypes of devices, including isochronous devices,can be controlled over IP networks.

Network File System (NFS) 15) is widely usedto share storage devices between computers.Since NFS is implemented using the virtual filesystem (VFS) layer in the UNIX operating sys-tem, it is not able to share either the commonAPIs for block devices nor the native I/O op-erations for ATA or SCSI disks, both of whichare lower-level functions than the VFS. For ex-ample, the NFS protocol does not define meth-ods to format a remote storage device, or toeject a remote removable media device. How-ever, USB/IP provides all the functions of a re-mote shared device because it is implementedin the lowest layer of an operating system.

University of Southern California’s Netsta-tion 4)∼6) is a heterogeneous distributed sys-tem composed of processor nodes and network-attached peripherals. The peripherals (e.g.,camera, display, emulated disk, etc.) are di-rectly attached to a shared 640 Mbps Myrinetor to a 100 Mbps Ethernet. The goal of Netsta-tion is to share resources and improve systemconfiguration flexibility. In this system, VISA(Virtual Internet SCSI Adapter) 12) emulatesdisk drives using UDP/IP. This project is simi-lar to our proposed system; both systems use IPnetwork to transfer peripherals’ data 8). How-ever, while Netstation studied network-basedcomputer architecture and substitutes existingsystems, our system aims to share already-attached devices between heterogeneous com-puters and proposes the most practical ap-proach in existing operating systems and ap-plications with exploiting today’s sophisticatedperipheral buses and their device drivers.

The Inside Out Network’s AnywhereUSB 9)

is a network-enabled USB hub. By their pro-prietary USB over IP technology, it providesremote access to the USB devices attachedto its USB ports in a quite limited manner.It supports only USB Bulk and Interrupt de-vices within 12 Mbps mode under LAN envi-ronments; most USB storage devices, whichare now implemented as 480 Mbps devices, andUSB isochronous devices are out of its target.Our USB/IP supports all types of USB de-vices with 480 Mbps mode. Our evaluation has

shown that in LAN environments all the deviceswork perfectly with the original PDD. More-over, the optimization strategy is also presentedto utilize our USB/IP more effectively even un-der larger network delays.

As one of the applications of USB/IP, itis possible to develop device switching soft-ware like existing KVM (Keyboard, Video, andMouse) switches. These switching applianceshave a virtual USB keyboard and mouse insideto always emulate these attachments to com-puters. This approach is also appropriate to thedevice switching software of USB/IP. USB/IPcan provide all kinds of USB devices with fairlydirect access over IP networks, which are nowdeployed everywhere by diverse media. Thisadvantage has enormous potentialities for use-ful applications. A practical application of theUSB/IP technology is now in progress to sharea device more easily between computers.

Wireless USB 18), which is under develop-ment, employs UWB (Ultra Wide Band) to ex-pand the USB connectivity. This technologyaims to eliminate USB cables. The communi-cation range is limited within 10 meters. Theimplementation of Wireless USB is almost inthe physical layer of USB. Therefore, this tech-nology is a complement for our USB/IP; ourUSB/IP will be also used for Wireless USB de-vices. Our USB/IP provides remote device ac-cess with IP network infrastructures which aredeployed everywhere.

5. Conclusion

We designed and implemented USB/IP as atransparent device sharing mechanism. We canuse various remote USB devices from existingapplications without any modification of the de-vice drivers. In our LAN, the I/O performanceof remote USB devices is sufficient for actual us-age. By the experiments, we showed the char-acteristics of USB/IP and the optimization cri-teria for IP networks.

To transfer fine-grained device control op-erations over IP networks, three design crite-ria should be considered carefully for the ap-plied networks. First, for asynchronous devicesas known as bulk devices, the device driversshould enqueue enough request data to getthe maximum throughput for remote devices.Second, for synchronous devices as known asisochronous devices, the smoothness of I/O isrequired. The appropriate number of requestsshould be enqueued to avoid the starvation of


requests on the device side. However, largequeuing size makes the responses of each de-vice worse. Last, the temporal restrictionsof the request responses should be considered.In greater or lesser degrees, all the requestshave their own temporal restrictions of the re-sponses. As far as the system works properly, itis possible to relax the temporal restrictions inthe device drivers or the applications and alsopossible to keep the buffering size minimum insynchronous transfers.

In future work, we continue to improve theUSB/IP technology to support various networkenvironments efficiently, such as a wireless net-work and a WAN.

6. Availability

The USB/IP implementation is available un-der the open source license GPL. The informa-tion is at http://usbip.naist.jp/.

References

1) Carson, M. and Santay, D.: NIST Net: ALinux-Based Network Emulation Tool, ACMSIGCOMM Computer Communication Review,Vol.33, No.3, pp.111–126 (2003).

2) Coker, R.: Bonnie++, http://www.coker.com.au/bonnie++/.

3) Cypress Semiconductor Corporation: EZ-USBFX2, http://www.cypress.com/.

4) Finn, G.G.: An Integration of Network Com-munication with Workstation Architecture,ACM SIGCOMM Computer CommunicationReview, Vol.21, No.5, pp.18–29 (1991).

5) Finn, G.G., Hotz, S. and Meter, R.V.: NVDResearch Issues & Preliminary Model (1995).

6) Finn, G.G. and Mockapetris, P.: NetstationArchitecture: Multi-Gigabit Workstation Net-work Fabric, Proc. Interop Engineers’ Confer-ence (1994).

7) Hirofuchi, T., Kawai, E., Fujikawa, K. andSunahara, H.: USB/IP — A Peripheral BusExtension for Device Sharing over IP Network,Proc. FREENIX Track: 2005 USENIX AnnualTechnical Conference (2005).

8) Hotz, S., Meter, R.V. and Finn, G.G.: Inter-net Protocols for Network-Attached Peripher-als, Proc. Sixth NASA Goddard Conference onMass Storage Systems and Technologies in Co-operation with Fifteenth IEEE Symposium onMass Storage Systems (1998).

9) InsideOut Networks: AnywhereUSB, http://www.ionetworks.com/.

10) Jini: http://www.jini.org/.11) Kroah-Hartman, G.: usbview, http://sf.net/

projects/usbview/.

12) Meter, R.V., Finn, G.G. and Hotz, S.:VISA: Netstation’s Virtual Internet SCSIAdapter, ACM SIGOPS Operating System Re-view, Vol.32, No.5, pp.71–80 (1998).

13) Nagle, J.: Congestion Control in TCP/IP In-ternetworks, http://www.ietf.org/rfc/rfc896.txt (1984).

14) Satran, J., Meth, K., Sapuntzakis, C.,Chadalapaka, M. and Zeidner, E.: InternetSmall Computer Systems Interface (iSCSI),RFC3720 (2004).

15) Sun Microsystems, Inc.: NFS: NetworkFile System Protocol Specification, RFC1094(1989).

16) Universal Plug and Play: http://www.upnp.org/.

17) USB Implementation Forum, Inc.: http://www.usb.org/.

18) Wireless USB Promoter Group: http://www.usb.org/wusb/home/.

(Received January 24, 2005)(Accepted May 6, 2005)

Takahiro Hirofuchi receivedthe B.S. degree in Geophysicsfrom Kyoto University in 2001,and the M.E. degree in Infor-mation Systems from Nara In-stitute Science and Technology(NAIST) in 2004. From 2004, he

has been a Doctoral Researcher of the 21st Cen-tury COE Program at the Department of In-formation Science of NAIST. Currently, he is aPh.D. candidate at NAIST. His research inter-est includes operating systems, networking, andnetwork services. He received the Best PaperAward of the USENIX 2005 Annual TechnicalConference. He is a member of IPSJ, USENIXand ACM.


Eiji Kawai received the B.S.degree in Mathematics fromKyoto University in 1996, andthe M.E. and D.E. degrees inInformation Systems from NaraInstitute of Science and Technol-ogy (NAIST) in 1998 and 2001,

respectively. From 2000 to 2003, he was anawarded researcher at Japan Science and Tech-nology Corporation. Since 2003, he has beenan assistant professor in NAIST. His researchinterests cover various topics relevant to dis-tributed computing environments, such as net-work protocols, information services, program-ming languages, system design, and perfor-mance evaluation. He is a member of ACM,IEEE, IPSJ, IEICE, and JSSST.

Kazutoshi Fujikawa is anAssociate Professor in GraduateSchool of Information Science,Nara Institute of Science andTechnology since 2002. He wasa visiting researcher of the Mul-timedia Communications Labo-

ratory at Boston University from March 1996through January 1997. He has been engagedin the project of bio/IT related R&D called“BioGrid Project,” which consists of several in-stitutes in Kansai area of Japan. His researchinterests widely cover distributed systems andmultimedia systems. Now he is very interestedin Grid systems for scientific simulations. Hereceived the M.E. and Ph.D. degree in informa-tion computer sciences from Osaka Universityin 1990 and 1993, respectively. He is a memberof IEICE, IEEE, and ACM.

Hideki Sunahara receivedthe B.S. and M.S. degrees inelectrical engineering from KeioUniversity in 1983 and 1985,respectively. He received thePh.D. in computer science fromKeio University in 1989. Cur-

rently, he is a professor in the Information Tech-nology Center, Nara Institute of Science andTechnology, Ikoma, Japan. His research fo-cuses on multimedia communication systems,digital libraries, computer architecture, parallelprocessing, distributed systems, operating sys-tems, and computer networks. He is a memberof ACM, IEEE, Internet Society, JSSST, andIPSJ. He is also a board member of the WIDEproject of Japan.

usb/ip: a transparent device sharing technology over ip

Documents