epics 2011 spring collaboration meeting, hsinchu, june 13-17, 2011 pcigeneral pci device support of...

EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011

pciGeneralPCI Device Support of EPICS

which is used in TPS Control System

Presented by: Jenny ChenTPS Control Group

June 14, 2011

2EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011

Outline

Goals of the Device Support –

pciGeneral

Rules of the Kernel Driver for PCI Cards

Details of the Kernel Driver Programs

Details of the Device Support Programs

Performance Tests

Access TLS control system via EPICS


Goals of the pciGeneral Programs

Manipulate as many PCI cards as possible

Manipulate the relevant resources

Handle interrupts

Convert byte order (big to little endian) if

needed

Keep maintenance programs to a minimum


Manipulate as Many PCI Cards as Possible Generalized kernel driver + pciGeneral + asynDriver(EPICS),

all PCI cards we have were tested successfully The tested PCI cards: cPCI-7452, 24DSI32, cPCI-EVG-300,

cPCI-EVR-300, TCP201 + DAC8402, TCP201 + DAC8415, TCP201 + ADC8417

Properties of PCI cards:– address space: IO-PORT or MEMORY

– DMA

– byte order: little-endian or big-endian

– daughter boards One kernel driver may handle several cards of the same model Special kernel driver (without accessing hardware): creates

shared memory – may exchange data between processes


Manipulate the Relevant ResourcesExtract 3 regions of the address spaces of a PCI card

Fully access the address spaces

plx -- the chip sets of PCI interface, ex: plx9000 series,

the mostly accessed registers are interrupt control and

status

card – includes the main control/status registers of the

PCI cards which are mostly for configuration

data – separated from the main registers (card), ex:

DMA, transient data or processed data (may be done by

the kernel driver or a running process)


Handle Interrupts

APs call read() to wait for the interrupts instead of

using ioctl() or signal(), which is easy to program

APs may directly access interrupt registers (plx), ex:

reset the registers when interrupts are not handled

properly

Kernel drivers serve enable or disable interrupts that

are requested by APs (ioctl)


Keep Maintenance Programs to a Minimum

Keep the software simple and small

Minimize the maintenance of IOC software

One kernel driver program per card + 2 device

programs– Ex: kernel driver: pci_adl7452.c for cPCI-7452– EPICS device support programs: pciGeneralDrv.c,

pciGeneralLinux.c


Rules of Kernel Driver Implementation

Create char device file in /dev, file name ex: pciCardName-x,

x: pci-slot, ex: pciadl7452-1, pciadl7452-2

ioctl() -- get the card information, enable/disable the interrupt,

setup/start/abort DMA, special service for the card

mmap() -- map the PCI address spaces into AP's

read() -- block AP to wait for an interrupt


Details of a Kernel Driver Program

Interface to a PCI card

Create a char device interface file

Serve char device access requests --

open/close/ioctl/mmap/read


Interface to PCI cardCreate char device interface file

#define MY_VENDOR_ID 0x144A

#define MY_DEVICE_ID 0x7452

#define SUB_VENDOR_ID PCI_ANY_ID

#define SUB_DEVICE_ID PCI_ANY_ID

#define MYCARD_NAME "pci7452"

static struct pci_device_id MY_DRIVER_PCI_TABLE[2] __devinitdata = {

{MY_VENDOR_ID, MY_DEVICE_ID, SUB_VENDOR_ID, SUB_DEVICE_ID, 0, 0, 0},

{0, 0, 0, 0, 0, 0, 0}

};

Declaration

static struct file_operations mypci_fops = {

.owner = THIS_MODULE,

.open = mypci_open,

.read = mypci_read,

.ioctl = mypci_ioctl,

.mmap = mypci_mmap,

.release = mypci_release,

};



Probe callback

- alloc_chrdev_region allocate char device resource, assign the char device name (should be unique)

- cdev_jnit

- cdev_add

- class_create create a class in directory /sys/class, ex: /sys/class/pci7452

- device_create create a device in /sys/class/device, ex: /sys/class/pci7452/pci7452-1

to insert/remove kernel module dynamic – according to a rule listed in /etc/udev/rules.d, the udev daemon sends

uevent to OS to create interface file: /dev/pciadl7452-1 (kernel version: linux2.6.13 or higher)

Create device interface file:

Request to access PCI address spaces- pci_request_regions request pci resources and associate with the device file: pci7452-1

- pci_resource_start get pci address spaces information

- pci_resource_end

- pci_resource_len

- pci_set_master to be the dma master

Initialize a waiting queue for interrupt request and variables

- init_waitqueue_head, atomic_set- kzalloc



Remove callback

- device_destroy destroy created device structure, ex: delete /sys/class/pci7452/pci7452-1

- class_destroy destroy created class structure, ex: delete /sys/class/pci7452

- cdev_del delete cdev structure

the char device interface file will be removed then

Delete device interface file:

Release PCI resources

- pci_release_regions release pci resources

- pci_disable_device release pci access

Free allocated memory

- kfree


Provide char device interface: open/close/ioctl/mmap/read

- open allow AP to access the char-device, call request_irq

- close release to access, call free_irq

- ioctl serve command: IOCTL_CARD_INFO, IOCTL_PLX_INFO, IOCTL_DATA_INFO,

IOCTL_INT_ENABLE, IOCTL_INT_DISABLE, IOCTL_DMA_SETUP, IOCTL_DMA_START,

IOCTL_DMA_ABORT, IOCTL_INITIAL, IOCTL_COMMAND

transfer PCI address information to AP, process interrupt requests, process DMA requests,

special service request to the kernel driver

- mmap map PCI address (memory type only) to user process address space: card, plx and data

- read serve interrupt request: block AP, wake up AP and copy interrupt count to AP

Serve the char device access requests


Details of the Device Support Programs for EPICS

They are based on asynDriver Programs:

– Header: pciGeneral.h

– pciGeneralDrv.c – the interface to EPICS, create records, register interrupt, parse arguments, ...

– pciGeneralLinux.c -- the interface to PCI cards, open/close device file, mmap pci address spaces, access the io-port/memory and create interrupt threads (call read() to wait)


EPICS Record types of pciGeneral

bi/bo/mbbi/mbbo ai/ao/longin/longout Waveform with DTYP: asynInt32ArrayIn/Out,

asynFloat32ArrayIn/asynFloat32ArrayOut asynFloat64ArrayIn/asynFloat64ArrayOut


DTYP: asynUInt32Digital for bi/bo/mbbi/mbbo

. bi/mbbi INP: @asynMask(portName,byteOffset,bitMask,timeout) command location

command: input/input16 location: plx/card/data

. bo/mbbo OUT: @asynMask(portName,byteOffset,bitMask,timeout) command location

command: output/output16/write/write16 location:

plx/card/data

(output/output16 – get readback first)

. bo (special) OUT: @asynMask(portName,0,0,timeout) command location actionState

command:

intEnable/intDisable/initial/dmaSetup/dmaStart/dmaAbort location: card

actionState: 0/1/-1 (0/1:process when state is 0/1, -1:process every

time)

INP/OUT Field of EPICS Record

record(bi,"diBank0Ch0") {

field(SCAN, “I/O Intr”)

field(DTYP,"asynUInt32Digital")

field(INP,"@asynMask(pci7452-1,0xc,0x1,1.0) input

card")

field(ZNAM,"low")

field(ONAM,"high")

}

record(bo,"doBank0Ch1") {

field(DTYP,"asynUInt32Digital")

field(OUT,"@asynMask(pci7452-1,0x0,0x2,1.0) output

card")

field(ZNAM,"low")

field(ONAM,"high")

}

appendix


Db example

record(mbbi,"erClockControlSts") { field(DTYP,"asynUInt32Digital") field(INP,"@asynMask(er230-2,0x050,0x0000ffff,1.0) input card") field(DESC, "Clock Control Status")}

record(mbbo,"setIntMaskBank0") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0x8,0xffffffff,1.0) write card")}

record(bo,"enableInt7452") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0,0,1.0) intEnable card 1") field(ZNAM,"null") field(ONAM,"enable") field(DOL, "clearIntBank0") field(OMSL, "1") field(HIGH, “0.1”) field(FLNK, "enableIntBank0")

}

st.cmd: pciGeneralDrvInit("tcp201-0","/dev/pcitcp201-0-IP"))

pciGeneralDrvInit("adc8417-0","/dev/pcitcp201-0-IPB"))

pciGeneralDrvInit("pci7452-1","/dev/pci7452-1")) pciGeneralDrvInit("er230-2","/dev/er230-2"))

appendix


DTYP: asynInt32 for ai/ao/longin/longout

. ai/longin INP: @asyn(portName,byteOffset,timeout) command location extractMask

command: input/input16, location: plx/card/data, extractMask: extract

several bits of 32/16 bits

. longin (special) INP: @asyn(portName,0,timeout) intCount card

. ao/longout OUT: @asyn(portName,byteOffset,timeout) command location extractMask

command: output/output16/write/write16/ioctl, location: plx/card/data

extractMask: extract/update several bits of 32/16 bits

INP/OUT Field of EPICS Record

record(longin,"diBank0") {

field(SCAN,"I/O Intr")

field(DTYP,"asynInt32")

field(INP,"@asyn(pci7452-1,0xc) input card 0xffffffff")

}

record(longout,"erPrescaler0Set") {


field(OUT,"@asyn(er230-2,0x100) output card 0x0000ffff")

}

record(longin,"intCountEg230") {

field(SCAN,"I/O Intr")


field(INP,"@asyn(eg230-3,0x0) intCount

card")

}

Ex: request kernel driver service: Ioctl card codecode parameterCode may be the pv value

record(longout,"$(IOC)-adc8417-$(S):ioctlCmd") { field(DTYP,"asynInt32") field(OUT,"@asyn(adc8417-$(S),0x0) ioctl card 0 $(HNUM)") field(DOL, "$(IOC)-adc8417-$(S):dataProc") field(OMSL, "1") field(FLNK, "$(IOC)-adc8417-$(S):ch0WfProc")}

appendix


DTYP: asynInt32ArrayIn/asynInt32ArrayOut for waveform

. waveform input

INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset

command: inArray/inArray16, location: card/data

. waveform output

INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset

command: outArray/outArray16, location: card/data

IncreaseAddrOffset depend on the data layout:

typeA ch0ch0ch0... one data block for one channel

(addrOffset:2/4)

ch1ch1ch1...

typeB ch0ch1ch2... one data block for N channels

(addrOffset:2N/4N)

INP/OUT Field Format of EPICS Record

record(waveform, "dac8402ch0Wf") {

field(DTYP,"asynInt32ArrayIn")

field(INP,"@asyn(dac8402-1,0x0) inArray16 data 0xffff

2")

field(NELM, "2048")

field(FTVL, "LONG")

field(PINI, “YES”)

}

record(waveform, "dac8402ch0WfOut") {

field(DTYP, "asynInt32ArrayOut")

field(INP,"@asyn(dac8402-1,0x0,1) outArray16 data 0xffff

2")

field(NELM, "2048")

field(FTVL, "LONG")

field(FLNK, "dac8402ch0Wf")

}

appendix


DTYP: asynFloat32ArrayIn/asynFloat32ArrayOut for waveform

INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset factor

offset

input waveform command: inArray/inArray16, location: card/data

output waveform command: outArray/outArray16, location: card/data

IncreaseAddrOffset: depend on the data layout

INP/OUT Field Format of EPICS Record

record(waveform, "dac8402ch0WfE") {

field(DTYP,"asynFloat32ArrayIn")

field(INP,"@asyn(dac8402-1,0x0) inArray16F data 0xffff 2 3.051e-4 -

10")

field(NELM, "2048")

field(FTVL, "FLOAT")

}

Refer to: http://www.icg.nsrrc.org.tw/wiki/EPICS_Drivers

appendix


Performance Tests

int_value = *((int *)mapped_ptr ;

vs.

ioctl(fd, IOCTL_READTEST, &int_value) ;

- Memory access reduces about 25% CPU time compared to ioctl

- Interrupt service test – EPICS client:

get the 5ms interrupt counter by LabCA (matlab)

ADLINK cPCI-6910:

CPU:dual-core Intel Xeon 2GHz,

cPCI:64bit/66MHz

Jitters:. OS scheduler. Dmc controller + network


Interrupt Service Test

- Interrupt test: the response time of a 5ms interrupt triggered by a

network process (measured by an AP)

Shared memory

kernel driver

char dev file: /dev/dmc4000-0

Data receiver

(from DMC4000)

Ethernet

as field bus

DMC4000 controller

send data

UDP

at every 5ms

Prepare for insertion devices control:

the field compensation by setting

correction coils (setting follows gap/phase)

needs to be done as fast as possible.

Wake up(read)

Ethernet kernel driver

(socket interface UDP)

open/mmap

ioctl

Control NetworkEPICS

ioc

EPICS ClientLabCA

open/mmap


Interrupt Test with cPCI-7452 DIO Card- Hardware interrupt test: connect a TTL signal to a DI channel, the state change would trigger the

PCI interrupt, the figures show the response time of 100Hz and 200Hz signal; the reason of some missing changes was unknown but it was probably not caused by the OS scheduler (maybe the interrupts were not delivered)

cPCI-7452 kernel driver

char dev file: /dev/pci7452-0

Test Program

open/mmap/read

Test condition:1. hi/low signal connect to ch02. testing loop: 500003. test process priority adjusted

cPCI-7452 DI Card

Wake up

Interrupt

200Hz/100Hz

ADLINK cPCI-6910:


cPCI:64bit/66MHz


EPICS ioc and LabCA Monitor Test- update test: the ioc updates data at every 5ms, the interrupts trigger

the ioc to update the dmc4000 and cPCI-7452 DIO channels

Shared memorykernel driverchar dev file:

/dev/dmc4000-0

Receive data

sent by DMC4000

Private Ethernet

DMC4000 controller send data

UDPat every 5ms

Wake up

Ethernet kernel driver

open

mmap

Ethernet

EPICS ioc

EPICS ClientLabCA

cPCI-7452 DI Card

Interrupt

Figure 1: ioc CPU load: 8%(top/Irix), db record: about 40

Figure 2: ioc CPU load: 20%(top/Irix), db record: about 170

Note: in figure 2, 128 DI

records scaned but not

in figure 1

kernel driver

char dev file:

/dev/pci7452-0

Wake

up

100Hz

ADLINK cPCI-6910:


cPCI:64bit/66MHz


Access Existing TLS Control System via EPICS Modify pciGeneral to tlsControl, because the data byte order

of TLS control system is not standard Start up a softIOC(+tlsControl) on a TLS console as a

gateway A simple char device driver (kernel) allocates memory to store

DDB which is updated by TLS ILCs and serves file operations: open/read/ioctl/mmap

TLS DDB-receiver flushes the DDBs into the memory which is also mapped by the softIOC, then two control systems access the DDB simultaneously

With the kernel driver running, the softIOC may be waked up by the interrupt which is triggered by the DDB receiver in order to update data synchronously

To control TLS devices from EPICS, streamDevice is used, a server waits for the string commands (name+value) then converts it (access SDB of TLS) to the format that ILC accepts


TLS Network

TLS ILCs

TLS control

software on consoleEPICS

IOC

Shared memorykernel driverchar dev file:

/dev/ilc-x

Setting server for

EPICSStreamDeviceTCP socket interface

TLS setting functions

TPS Network

File interface mmap()ioctl()File interface

mmap()read()

TLS SDB/DDB

The Conjunction for TPS and TLS Control System

10Hz DDB

On-demand Setting

Wake up

DDB Upload

Setting server

TLS ILC control

system

Run EPICS ioc on an ILC of TLS control system

TLS ILC

kernel driverchar dev file

TLS Network

EPICS IOC

TCP socket

Access any device of TLS control system via EPICS

TLS Console

Setting server (ca-lib) for ILC

IO hardware


Summary

• Efficiency

• Flexibility

• Maintenance

EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011

Acknowledgement

Marty Kraimer

Thank you for your attention!

epics 2011 spring collaboration meeting, hsinchu, june 13-17, 2011 pcigeneral pci device support of...

Documents