the ccp4 package
DESCRIPTION
The CCP4 Package. General Overview. On What Systems Will It Work?. SGi: Irix (o32, n32 and 64bit) Linux Systems (Redhat, SuSE…..) Compaq OSF1 etc Sun (Sun OS) HP (All apart from Xwindows) IBM (AIX) Mac OSX Microsoft Windows +. Where It Can Be Downloaded. CCP4 ftp site: - PowerPoint PPT PresentationTRANSCRIPT
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
1
General Overview..
The CCP4 Package.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
2
On What Systems Will It Work?SGi: Irix (o32, n32 and 64bit)Linux Systems (Redhat, SuSE…..)Compaq OSF1 etcSun (Sun OS)HP (All apart from Xwindows)IBM (AIX)Mac OSXMicrosoft Windows+
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
3
Where It Can Be Downloaded CCP4 ftp site:
ftp://ftp.ccp4.ac.ukftp ftp.ccp4.ac.ukuser name: ftppw [email protected] pub/ccp4/current/binary
CCP4 mirror sites:Japan
ftp://ftp.protein.osaka-u.ac.jp/mirror/ccp4 USA
ftp://ftp.sdsc.edu/pub/sdsc/xtal/CCP4
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
4
How BIG Is the Download?
unpacked size =160Mb tar.gz size = 36 Mb tar.Z size = 56 MbBinary size = 25 – 107 Mb
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
5
InstallationBinaries or source code?UnpackingDirectory structureccp4.setupConfigureSource Code MakeBinary MakeWindows Installing on multiple systems
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
6
Installation IIYou do not need root privileges on UNIX systems
Just enough space!You DO need Admin privileges on Windows NT/2K
This is used to set up environment variables etc.
What else do you need to install CCP4?Source code installation:
Compilers C and FORTRAN.Make (UNIX)
Gzip/Uncompress/tar (UNIX)ccp4i requires tcl/tk/BLTAstexViewer requires a Java VM.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
7
Binaries or Source Code?
Binary Source
Install (inc ease) speed Download speed Developers (Customisation) Range of systems/portability Compiler problems/price Ease of support
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
8
Binaries or Source II?
SupportedBinaries?
yes
no
Developer?FastNet?
SourceCode andCompile
yes
no
ReliableCompiler?
yes
no
yes
Binary+
Source code
no
(Currently supported: irix, linux, OSF1, SunOS, Windows)Binaries
or source?
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
9
Unpacking
Download the file into an area with enough space to download and unpack it.To uncompress a .gz file use gunzipTo uncompress a .Z file use uncompressA tar file is a collation of many files: To extract the files from a tar file use:
tar –xvf file.tar
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
10
Directory Structure
ccp4_version.tar xtal/ccp4_version/
ccp4i etc include lib src unsupported x-windows
doc, examples, help, htm
l, man, m
anual
CC
P4 interface, Interface
documentation, roadm
aps
CC
P4 utility scripts
ccp4.setup, environ.def and default.def
Library src and data
Program
Source C
ode
bin
Older program
s that are being phased out
Program
s with
graphical user interfacethat require x-w
indows
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
11
ccp4.setup
All CCP4 programs require ccp4 environment variables set-up to enable them to run.
ccp4.setup files found in include sub directoryccp4.setup-dist and ccp4.setup-bashIt is essential that the following environment variables are
correctly assigned:CCP4_MASTER (where the ccp4_vers.tar was unpacked)CCP4_SCR (CCP4 scratch directory), CCP4I_TCLTK (where ccp4i finds bltwish)
Most other environment variables use these variables.source ccp4.setup before configure and running CCP4 programsAutomatically source ccp4.setup in login files (.cshrc or .login)
(UNIX variants only)
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
12
ccp4.setup II
BINSORT_MEMSize of memory to use for sorting before suing disk
CCP4I_HELP Usually $CCP4/ccp4i/help
BINSORT_SCR Disk area to use for sorting CCP4I_TCLTK Which bltwish to use
CBIN Where ccp4 binaries will live CCP4I_TOP Usually $CCP4/ccp4i
CCP4 Top directory of ccp4 CHTML $CCP4/html
CCP4_BIN $CBIN CINCL $CCP4/include
CCP4_LIB Where ccp4 libraries will live CLIBD $CCP4/lib/data
CCP4_MASTER Usually $CCP4/../ CLIBS $CCP4/lib/src
CCP4_OPENOverwrite files or not.
UNKNOWN = overwriteCPROG
$CCP4/src home of the ccp4 program code
CCP4_SCRWhere ccp4 programs will write temporary files.
PUBLIC_FONT84$CCP4_LIB/font84.dat a font data file.
Environment variables
There is no need to learn these (and there are more) unless things go wrong!
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
13
configure
Configuring allows system specific flags to be assignedConfigure gives you flexibility, and a quick and easy
way to change all the important make files Configure settings can be overridden by environment
variablesconfigure + Makefile.in Makefile + config.status
Once the system has been configured you can either make the source code or install the binary files.
A log of the configure settings are in config.status .
More than one configure is run by $CCP4/configure
(UNIX variants only, when making from source code)
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
14
configure (II)configure [--flags] [operating system]
Operating system
(choice of 17!)
--flags
(choice of 15!)meaning
irix, irix64 SGi --with-xCompile the X
programslinux
/_compaq_compilersLinux --{*}dir src/bin/lib/tmp
hpuxHewlett Packard
--with-shared-lib Compile shared libs
osf1 Compaq --only-libs Only compile libs
Darwin Mac OSX --with-warnings One for the developers!
sunosSun
Microsystems
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
15
Source Code Make(installing from source code only)
Once you’ve downloaded the package, unpacked, set the environment variables and configured the package you are now ready to make the suite!
In $CCP4 type make.You might want to keep a log with make > make.log
Make will firstly make the library components before the programs.
Once complete you will need to install with make install.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
16
Source Code Make II.The make utility uses the settings in the Makefile which in turn was created from the Makefile.in by the configure script.
$CLIBS/fsplit – a small program used to break large library.f files into smaller files – this was needed on older systems.
Make in the x-windows directory.The make in the x-windows directory will ignore errors in compiling programs and will carry on regardless.
Once complete you will need to install with make install.This will install the programs into $CBINMoves the library files into $CCP4_LIB / $CLIB
Make also has numerous other options.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
17
Source Code Make IIImake [option]
Option Meaning
install Installs the components into the correct area
[program] Will make the program if its been updated
clean, realclean, distclean
Various degrees of severity of cleaning the directory structure.
uninstall What it says on the box…..
empty-targets Sets the size of e.g. $CPROG/[program] to zero
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
18
Source Code Make IV(reference only)
Make option
clean Removes *.o, *.~, etc
realcleanclean + any executable files in the directory
distcleanRealclean + removes Makefiles, config.status -> only distribution left
uninstallRemoves installed files created by the install step
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
19
Source Code Make V - Compilers
Under the majority of operating systems the native compilers should be used.
The GNU compilers gcc and g77 can used on many systemsRedhat 7.0 and 7.1 users should upgrade their gcc/g77Use gcc 3.0.<4
Linux users can also use a variety of other compilers.See recent ccp4bb conversations
(e.g Portland Group and Intel compilers).
Irix 6.5 users should not use f77 version 7.2.*We currently use 7.3.1.2m
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
20
Binary Code Make
You should have:$CCP4/BINARY.readme$CCP4/BINARY.setup$CCP4/bin/
You need to:Get normal distribution and unpack itEdit and source ccp4.setupRun the script BINARY.setup
See also http://www.ccp4.ac.uk/dist/INSTALL.html BINARY.about file in ‘binary’.tar.gz
(UNIX variants only)
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
21
WindowsTested on Windows 95/98/Me/NT(4.0)/2000You will need Admin privileges
Windows NT and 2000Use .exe auto install
Windows 95/98/Me variantsUse compressed .zip file
You will also need Tcl/tk and BLT for ccp4i (install before ccp4)
http://www.ccp4.ac.uk/dev/ccp4onnt.html
Cygwin is possible.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
22
Installing on Multiple SystemsMultiple systems means e.g. Linux and Irix with shared/mounted disks
Install completely independent source treeUse duptree (found in $CCP4)
duptree must be run before any files are altered in $CCP4duptree creates a second identical source tree using links
Use --srcdir
Why? Size matters. You only need to bug fix one src.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
23
Bug Fixing and Patching.
http://www.ccp4.ac.uk/problems.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
24
More Information
CCP4 manual.CCP4 website.
http://www.ccp4.ac.uk/dist/INSTALL.htmlCCP4 problems pages.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
25
Now What Have You Got?
The programs
How they communicate
CCP4 Data formats
How to use the programs
The main programs
What is new in 4.2
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
26
The ProgramsMost programs written in standard(ish) fortran77
Over 150 programs mostly runnable from command line
Full listings on:http://www.ccp4.ac.uk/dist/html/INDEX.html http://www.ccp4.ac.uk/dist/html/FUNCTION.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
27
How They Communicate?Standard data formats that contain informationand informative log outputs.
The data formats are publicly known, documented and useable by anyone.
Nobody is forced to use our data formats as wesupply numerous conversion tools.
Some CCP4 programs already output keyinformation in XML files.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
28
CCP4 Data Formats.Formats used are:
.mtz – reflection data.map .msk – map and mask formats.pdb – atom positions.cif – harvest files, refmac5 restraints
Binary formats were used for:SizeSecurity
MTZ/map files are binary but portable as they contain a machine stamp
More information in the FILE FORMATS section onhttp://www.ccp4.ac.uk/dist/html/INDEX.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
29
CCP4 Data Formats II – MTZ Reflection Files
Table of datacolumns - I, F, sigma, phase, FOM, etc. etc.rows - reflections
Columns grouped as datasets withproject name GerEdataset name se_inflection cell dimensions 108.7 61.6 71.6 90.0 97.1 90.0wavelength 0.981
In ccp4i, see:
Reflection Data Utilities -> Edit MTZ Project&Dataset
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
MTZ fileTitle/historySpacegroup
Crystal 1Crystal nameProject nameCell dimensions
Crystal 2Crystal nameProject nameCell dimensions
Dataset 1.1Dataset nameWavelength
Dataset 1.2Dataset nameWavelength
Column Column
Heirarchical view of reflection data.
CCP4 Data Formats III – MTZ Reflection Files II
The future ....
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
31
CCP4 Data Formats IV – MTZ Reflection Files III
Header information
Keyword – value pairs
TITLE – title NCOL – no. of data columns CELL – global cell parameters SYMINF – symmetry info COL – column information
RESO – resolution rangeNDIF – no. datasetsDATASET – dataset infoPROJECT – harvesting infoBATCH – batch images
TITLE make ccp4itut mtz inconspicuous Sfs from refined RNA_SANAT 8-7. NCOL 8 18077 0 CELL 64.8970 78.3230 38.7920 90.0000 90.0000 90.0000 SORT 1 2 3 0 0 SYMINF 4 4 P 19 P212121 PG222 COLUMN K H 0.0000 42.0000 COLUMN L H 0.0000 21.0000 COLUMN FP F 0.0000 1219.0000
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
32
CCP4 Data Formats V – MTZ Reflection Files IV
Viewing and manipulation
mtzdump – view ascii dump of mtz file
mtz2various – convert mtz file to MULTAN, SHELX, TNT, X-PLOR/CN* formats
mtzutils, cad, sftools – edit mtz files
scaleit, fhscal – column scaling
http://www.ccp4.ac.uk/dist/html/FUNCTION.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
33
CCP4 Data Formats VI – MAP Files - Densities
mtz map Fourierphases
fft –switch between electron density and reflection data
mapdump – give information on map
View map with – O, PyMol, Mapslicer, AstexViewer
Format – header + symops + data
Binary - but machine independent.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
34
CCP4 Data Formats VII – Atom Positions
PDB files:Based on pdb version 2.1 draft
mmCIFCCP4 uses a subset of the mmCIF dictionaryUsed in;
Restraints library of Refmac5 and libcheckmmdb libraryHarvesting files
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
35
How To Run The Programsccp4i – the ccp4 interface
Command line options:program [<logical name> <file name>]
e.g. fft hklin native-Fs.mtz mapout 2Fo—Fc.map
Keyworded input:Keyword <options>
E.g. CELL a b c [ ]TITLE this job will work this time
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
36
How To Run The Programs IICommon Keywords
CELL a b c [ ]Cell dimensions in Å
RESOLUTION <limit> [<limit>]
Resolution limits
ENDTerminates input and starts program
SYMMETRY
<number> | <name> | <operator>HISTORY
Adds record to history stack of MTZ
LABIN <Program label>=<file label>…Associates the column labels that the program expects with MTZ cols
LABOUT <Program Label>=<file label>..Associates column labels used in output file with those in program.
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
37
The Main Programs
What do you get?
>150 programs encompassing the majority of protein crystallography – including:
Mosflm, Scala, Truncate, Mlphare, Rantan, Revise, Acorn, Oasis, SC, Amore, Molrep, Beast, DM, Fffear, Refmac5, Procheck, Sfcheck
See http://www.ccp4.ac.uk/dist/html/INDEX.html or http://www.ccp4.ac.uk/dist/html/FUNCTION.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
38
What Is New in 4.2ACORN ab initio procedure for the determination of protein structure at atomic resolution (Yao Jia-Xing)BEAST Brute-force molecular replacement with Ensemble Average Statistics, Maximum likelihood-based molecular replacement (Randy Read) PROFESSS determination of NCS operators from heavy atoms (Kevin Cowtan) ROTAMER list amino acids whose side chain torsion angles deviate from Richardson's Penultimate Rotamer Library (Dirk Kostrewa) ASTEXVIEWER Java application for viewing proteins, ligands and electron density maps (Mike Hartshorn)
Announcement on: http://www.ccp4.ac.uk/news/msg00098.html
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
39
An Introduction to CCP4iThe CCP4 Graphical User Interface
Author: Peter BriggsCCP4
CCP4i Introduction
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
40
• Offers user friendly interfaces to the programs
• Integrated help system
• Tools for file viewing & “project management”
How to get started
• (Unix/Linux) Type ccp4i at the command prompt
• (Windows/NT) Select ccp4i via the Start menu
Why Use CCP4i?
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
41
Tasks
Modules
Job Database Tools & Utilities
On-line help
Main CCP4i Window
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
42
• Access the module list by clicking on the gold bar displaying the current module
Modules
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
43
• Tasks used in a particular part of the structure determination process• Utility tasks for manipulating different types of data• Alphabetical list of programs/tasks
Modules II
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
44
• Click on the appropriate button in the list to start a particular task
• This starts the interface for that task
Tasks
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
45
Closed foldersAdvanced/infrequently
used options
Open foldersParameters that should be checked
by the user before runningHighlights indicate compulsory input
File folderSet input and output file names
Protocol folderMake the key decisions
WO
RK
FR
OM
TH
E T
OP
DO
WN
Run taskSave/restore parameters
Always add a title to distinguish different runs of the same task
Defaults - “If it’s not visible then it’s not important”
Example Task Interface
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
46
Watch the progress of the job in the Job Database window:
Running a Task
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
47
• View logfile in Netscape (if it contains HTML tags) and/or text browser
• View input and output files (.mtz, .pdb, CCP4 maps) using appropriate viewer- also use the View Any File option)
- or ccp4i -v <filename> at the prompt
• View graphs in logfile using loggraph- also use loggraph <filename> at the command prompt
View Output From Job
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
48
• Why bother?
• Benefits:- remind you what you did six months ago- keep track of multiple projects and associated data- facilitate “back-tracking”- make it easier to deposit your results/write your paper
Project Management Using CCP4i
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
49
“All data files relating to one crystallographic project should be in a single project directory”
One word alias ...… for project directory containing data files
Project Directories
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
50
• Each project directory has an associated job database accessed through the central panel in the main window:
• This displays:- which tasks were run, and when- their status (RUNNING, FINISHED, FAILED etc)- the title entered by the user
• The database also keeps a record of:- the parameters used to run the task- the input, output and log files associated with the task
Job Database
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
51
View files from any job in the database
Remove failed/unwanted jobs from the database and archive important data
Rerun any job in the database (with the option of changing the parameters first)
• Use this to review parameters used in an earlier run
Keep the database up-to-date• Add runs of “external” programs
Job Database Utilities
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
52
Notebook: record information for future reference
Edit data: e.g. update title of run, change file locations
Add external /non-interfaced program run to the database
Edit Job Data
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
53
General help from main window
Help with a particular option -Right hand mouse button click
over that option
Help for a particular task
Brings up relevant documentation in browser
On-Line Help
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
54
• Message line help
One-line help message appears in the bar at the top of the window
• Send e-mail to directly to CCP4(please send as much information as possible!)
More Help….
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
55
• CCP4 Roadmaps - accessed via program index (under General)• CCP4 Tutorial2000 - also from the program index
Navigating Modules
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
56
•On-line help:- help button on main window for general info- help button on task interfaces for help with individual tasks- right-hand mouse button click over part of the window for help on that option or feature.
• Roadmaps through the structure determination process:- $CCP4/ccp4i/help/roadmaps/index.html
• Tutorial material- $CCP4/examples/tutorial2000/html/index.html
• E-mail CCP4- [email protected]
CCP4 Help Summary
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
57
• Key programs in the structure solution procedure write out harvest files- in CCP4 these are SCALA, TRUNCATE, MLPHARE, REFMAC and RESTRAIN
• At deposition time these files represent an accurate record of how the final model was obtained
• Each file records details of the method used and the results obtained (e.g. heavy atom sites used in phasing)
• Harvest files can be sent directly to the deposition site, avoiding much manual processing
Data Harvesting
CCP4 Workshop ACA, Texas 2002
presentation by Alun Ashton, [email protected]
58
• Harvesting operates “invisibly”
• Harvesting requires MTZ files contain Project and Dataset names:- add these when data is first imported into CCP4 (Import Unscaled Data or Convert to MTZ & Standardise tasks in Data Reduction Module), or- edit Project name or Dataset name using Edit MTZ Project&Dataset (in the Reflection Data Utilities module)
• Switch on the harvesting options:- in individual tasks e.g. SCALA, REFMAC, or- set the harvesting defaults in the CCP4i Preferences window (RHS of the main window)
- you don’t need to think about it again until the end of the structure solution
Data Harvesting in CCP4i