workshop matlab bits database
DESCRIPTION
Matlab Bank of Italy Time Series DatabaseWorkshop c/o Area Ricerca Banca d'ItaliaVilla Huffer21 Gennaio 2008TRANSCRIPT
WMDB 2008BITS Time Series Data Base
Emmanuele [email protected]
Supporto Informatico per l’Area RicercheBanca d’Italia
21 January 2008
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 1/28
Summary
1 tsdb The Time Series Database with Vintage SupportLoading DataGetting informations
2 tsdb Administration
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 2/28
tsdb The Time Series Database with Vintage Support
1 tsdb The Time Series Database with Vintage SupportLoading DataGetting informations
2 tsdb Administration
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 3/28
tsdb Time Series Database Object
Single object for all Database operations (tsdb)
Very simple command interface
Saves data, release date, series name and every metadata you wish
Issues (standard) SQL Data Manipulation commands to RDBMS.
Issue also Data Definition commands to fully create database instanceor grant access to user (db root authorizations required)
Fully support free MYSQL database, compatible to other DB (DB2and Oracle tested).
Native access to JDBC drivers (more speed) or ODBC Connectors(more compatibility)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 4/28
tsdb Basic Usage
>> db = tsdb( database, user, password)>> ts = tsload(db, ’SERIENAME’)>> [...]>> tsstore(db, ts)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 5/28
tsdb What is a Release?
A RELEASE is a value representing a lapse in time at your wish (i.e. whenyou downloaded the series).The field containing the release information is named release and is inthe columns metadata.Actual TSDB implementation recognize release as MATLAB datenumvalue.Even the container (tsmat) has a release metadata, it NOT reports therelease of the series contained but only the release of the collection instead.The series (column) release is saved into the DB with tstore operationand loaded with tsload whereas collection release is not saved andloaded. Only if isn’t defined a release for a series/column tsstoreoperation uses collection release as series release.
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 6/28
tsdb Collection and Series Releases
>>
tstab(ts)tsStart: 1980 1tsRelease: 25-Jul-2007
COLLECTION RELEASE
Frequency: 12 (monthly)Date
SERIES RELEASE
T1 T1 T1 T117/07/07 18/07/07 19/07/07 20/07/07
1980-M-001 0.6822 0.1509 0.8600 0.49661980-M-002 0.3028 0.6979 0.8537 0.89981980-M-003 0.5417 0.3784 0.5936 0.8216
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 7/28
tsdb Collection and Series Releases
>> tstab(ts)
tsStart: 1980 1tsRelease: 25-Jul-2007
COLLECTION RELEASE
Frequency: 12 (monthly)Date
SERIES RELEASE
T1 T1 T1 T117/07/07 18/07/07 19/07/07 20/07/07
1980-M-001 0.6822 0.1509 0.8600 0.49661980-M-002 0.3028 0.6979 0.8537 0.89981980-M-003 0.5417 0.3784 0.5936 0.8216
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 7/28
tsdb Collection and Series Releases
>> tstab(ts)tsStart: 1980 1tsRelease: 25-Jul-2007
COLLECTION RELEASE
Frequency: 12 (monthly)Date
SERIES RELEASE
T1 T1 T1 T117/07/07 18/07/07 19/07/07 20/07/07
1980-M-001 0.6822 0.1509 0.8600 0.49661980-M-002 0.3028 0.6979 0.8537 0.89981980-M-003 0.5417 0.3784 0.5936 0.8216
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 7/28
tsdb Collection and Series Releases
>> tstab(ts)tsStart: 1980 1tsRelease: 25-Jul-2007 COLLECTION RELEASEFrequency: 12 (monthly)Date
SERIES RELEASE
T1 T1 T1 T117/07/07 18/07/07 19/07/07 20/07/07
1980-M-001 0.6822 0.1509 0.8600 0.49661980-M-002 0.3028 0.6979 0.8537 0.89981980-M-003 0.5417 0.3784 0.5936 0.8216
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 7/28
tsdb Collection and Series Releases
>> tstab(ts)tsStart: 1980 1tsRelease: 25-Jul-2007
COLLECTION RELEASE
Frequency: 12 (monthly)Date SERIES RELEASE
T1 T1 T1 T117/07/07 18/07/07 19/07/07 20/07/07
1980-M-001 0.6822 0.1509 0.8600 0.49661980-M-002 0.3028 0.6979 0.8537 0.89981980-M-003 0.5417 0.3784 0.5936 0.8216
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 7/28
tsdb Using Single and Multiple Parameters
Some TSDB operations may be used with a single parameter as well withmultiple parameters.Single parameter could be a simple string, numbers or series. Multipleparameters have to be a cell array containing strings, numbers or series.
>> ts = tsload(db, ’SERIES’)>> ts = tsload(db, { ’SERIES1’ ’SERIES2’})>> ts = tsload(db, { ’SERIES1’ ’SERIES2’}, 733283)>> ts = tsload(db, { ’SERIES1’ ’SERIES2’}, {733283 733284})
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 8/28
tsdb A complete example
Follow instructions to setup free database software on your machine
Follow instructions to setup Java connector on your machine
Connect as database administrator and create a TSDB instance
Connect as BITS user and test installation
Play freely
Return to work
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 9/28
tsdb Setup free DBMS (Mysql)
Downloads and click on XAMPP.EXE(from file://stux20/pccommon/xampp)
Unzip in D:/
Done (Your administrator user is root. No password.)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 10/28
tsdb Setup Java
Downloads mysql-connector-java-5.0.6-bin.jar(from file://stux20/pccommon/xampp)
Save in D:/xampp/mysql/
>> javaaddpath D:/xampp/mysql
>> clear java
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 11/28
tsdb DEMO: Create demo database
From Matlab prompt issue the commands:
>> dbROOT = tsdb(’J:mysql’,’root’,’’)
>> tsdbdemo(dbROOT,’create’)
>> close(dbROOT)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 12/28
tsdb DEMO: Check database
>> db = tsdb(’J:bitsdemo’,’BITS’,’’)BITSDB Database bitsdemo (ver. 1.0)User: BITS
On Host: localhostBackup Dir: C:\Documents[...]\h856605\Application Data\BITSDB
Driver: jdbc:mysql://localhost/bitsdemo
>> namelist=tsseries(db)
namelist = ’ITRETTOTF’’ITIPINTMG’’BDESPISDH’’BDEU2001A’’BDM3C...B’’FRHCONMGD’’EAGDP...D’’ITGDP...D’
OK. Database ’bitsdemo’ is up and running.E. Somma (SIA-BdI) WMDB 2008 21/01/2008 13/28
Loading Data
1 tsdb The Time Series Database with Vintage SupportLoading DataGetting informations
2 tsdb Administration
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 14/28
tsdb DEMO: Loading Data
>> [ tsM, tsA ] = tsload(db,namelist)tsM =
matdata: [258x6 double]start_year: 1986
[...]Size: [258 6]Start: 1986 1
End: 2007 6Labels: ITRETTOTF ITIPINTMG BDESPISDH BDEU2001A BDM3C...B
tsA =matdata: [89x2 double]
start_year: 1985[...]
Size: [89 2]Start: 1985 1
End: 2007 1Labels: EAGDP...D ITGDP...D
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 15/28
tsdb DEMO: Loading Data/2
>> tstab(tsM)tsStart: 1986 1tsRelease: 19-Jul-2007Frequency: 12 (monthly)Date
ITRETTOTF ITIPINTMG BDESPISDH BDEU2001A BDM3C...B FRHCONMGD12/07/07 12/07/07 12/07/07 12/07/07 12/07/07 12/07/07
1986-M-001 39.1000 78.8519 82.3500 22398.1600 601.8000 12.07801986-M-002 35.7600 79.5136 95.3100 21864.3700 602.9000 11.90101986-M-003 42.5700 79.4736 101.4700 22359.8200 605.3000 12.23301986-M-004 41.1700 79.2002 96.9700 25555.3900 607.4000 12.4070[...]2007-M-004 106.6000 95.8000 91.1000 78810.0000 1700.0000 20.87702007-M-005 NaN 96.6000 91.3000 79077.0000 1711.8000 20.71102007-M-006 NaN NaN NaN NaN NaN NaN
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 16/28
tsdb DEMO: Loading Data/2
>> tstab(tsM)tsStart: 1986 1tsRelease: 19-Jul-2007Frequency: 12 (monthly)Date
ITRETTOTF ITIPINTMG BDESPISDH BDEU2001A BDM3C...B FRHCONMGD12/07/07 12/07/07 12/07/07 12/07/07 12/07/07 12/07/07
1986-M-001 39.1000 78.8519 82.3500 22398.1600 601.8000 12.07801986-M-002 35.7600 79.5136 95.3100 21864.3700 602.9000 11.90101986-M-003 42.5700 79.4736 101.4700 22359.8200 605.3000 12.23301986-M-004 41.1700 79.2002 96.9700 25555.3900 607.4000 12.4070[...]2007-M-004 106.6000 95.8000 91.1000 78810.0000 1700.0000 20.87702007-M-005 NaN 96.6000 91.3000 79077.0000 1711.8000 20.71102007-M-006 NaN NaN NaN NaN NaN NaN
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 16/28
tsdb DEMO: Loading Releases
>> r = tsreleases(db,’BDESPISDH’);>> ts = tsload(db,’BDESPISDH’,r)ts =
matdata: [258x25 double][...]
Size: [258 25]Start: 1986 1End: 2007 6
Labels: BDESPISDH BDESPISDH ...
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 17/28
tsdb DEMO: Loading Data/2
>> tstab(ts)tsStart: 1986 1tsRelease: 06-Sep-2007Frequency: 12 (monthly)Date
BDESPISDH BDESPISDH BDESPISDH28/02/07 08/03/07 09/03/07
1986-M-001 82.3500 82.3500 82.35001986-M-002 95.3100 95.3100 95.31001986-M-003 101.4700 101.4700 101.4700[...]2006-M-012 89.3000 89.3000 91.60002007-M-001 NaN NaN 91.20002007-M-002 NaN NaN NaN2007-M-003 NaN NaN NaN
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 18/28
Getting informations
1 tsdb The Time Series Database with Vintage SupportLoading DataGetting informations
2 tsdb Administration
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 19/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}
== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}
== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}
== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}
== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)
== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}
== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)
== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}
== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}
== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}
== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}
== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
tsinfo
res = tsinfo(db,series,[operation, [arg, ... ] ] ...)op result = res{#idx}
== NSERIES - Number of series in DB ==>> res = tsinfo(db, nan, ’nseries’)>> nseries = res{2}== SERIES - Names of the series in DB ==>> series = tsinfo(db,nan, ’series’)== NRELEASES - Number of releases in series ==>> res = tsinfo(db, {series}, ’nreleases’)>> nrels = res{1,1}>> nerls_serie1 = nrels{1}== RELEASES - Set of release dates for a given series ==>> rels = tsinfo(db, {series}, ’releases’)>> rels_serie1 = rels{1}
== ORDERBYFREQ - returns series ordered by frequency ==>> [ordered, freqs] = tsinfo(db, {series}, ’orderbyfreq’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 20/28
Connection status
>> ping(db)DatabaseProductName: ’MySQL’
DatabaseProductVersion: ’5.0.41-community-nt’JDBCDriverName: ’MySQL-AB JDBC Driver’
JDBCDriverVersion: [1x103 char]MaxDatabaseConnections: 0
CurrentUserName: ’BITS@localhost’DatabaseURL: ’jdbc:mysql://localhost/bitsdemo’
AutoCommitTransactions: ’True’
>>isconnection(db)ans =
1
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 21/28
Connection status
>> ping(db)DatabaseProductName: ’MySQL’
DatabaseProductVersion: ’5.0.41-community-nt’JDBCDriverName: ’MySQL-AB JDBC Driver’
JDBCDriverVersion: [1x103 char]MaxDatabaseConnections: 0
CurrentUserName: ’BITS@localhost’DatabaseURL: ’jdbc:mysql://localhost/bitsdemo’
AutoCommitTransactions: ’True’
>>isconnection(db)ans =
1
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 21/28
tsdb Administration
1 tsdb The Time Series Database with Vintage SupportLoading DataGetting informations
2 tsdb Administration
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 22/28
tsadmin(dbROOT,’setup’,...)
Create USER if doesn’t exists.Requests password interactivelyor use empty password if DBNAME begins with ’*’
Create database bits<DBNAME>
Setup TSDB structure in ’bits<DBNAME>’
Example:
>> tsadmin(dbROOT,’setup’, ’BITS’,’test’)>> tsadmin(dbROOT,’setup’,’*BITS’,’test’)
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 23/28
tsadmin(dbROOT,’drop’,DBNAME)
Prerequisite:
Administrator account/password (root/’’)
Operations:
Expunge db instance
Needs interactive confirmationUnless variable ’force’ is defined in work area
Example:
>> dbROOT = tsdb(’J:mysql’,’root’,’’);>> tsadmin(db,’drop’,’bitstest’);
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 24/28
tsadmin(db,’trunc’)
Prerequisite:
User account/password (BITS/’’)
Operations:
Clean all db tables contents
Needs interactive confirmationUnless variable ’force’ is defined in work area
Example:
>> db = tsdb(’J:bitsdemo’,’BITS’);>> tsadmin(db,’trunc’);
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 25/28
Backup/Restore operations
Backup/restore operations save/load data in BACKUPDIRSave full time-series database, you can use fullbackup.
>> tsadmin(db,’fullbackup’,LABEL);It save a text file named where <date> is today date in’dd-Mmm-yyyy’ format.
Save all releases of a single series, use tsbackup.>> tsadmin(db,’tsbackup’,LABEL);
It save a matlab .mat file named <SERIENAME> <date> <LABEL>.matwhere <date> is today date in ’dd-Mmm-yyyy’ format.
Operations:Clean all db tables contentsNeeds interactive confirmation
Unless variable ’force’ is defined in work area
Example:
>> db = tsdb(’J:bitsdemo’,’BITS’);>> tsadmin(db,’trunc’);
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 26/28
tsadmin(db,’rm’, { series })
Prerequisite:
User account/password (BITS/’’)
Operations:
Deletes time-series and (part of) his history from db
Example:
>> db = tsdb(’J:bitsdemo’,’BITS’);>> tsadmin(db,’rm’, ’BDESPISDH’);>> tsadmin(db,’rm’, ’BDESPISDH’,
{datenum(’28/02/2007’)});
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 27/28
tsadmin(db,’cp/mv, { series }, db2)
Prerequisite:
User account/password (BITS/’’) on db1 and db2
Operations:
Copies or Moves a time-series and (part of) his history from one db toanother
Example:
>> db = tsdb(’J:bitsdemo’,’BITS’);>> db2 = tsdb(’J:bitsdemo2’,’BITS’);>> tsadmin(db,’cp’, ’BDESPISDH’, db2);>> tsadmin(db,’mv’, ’BDESPISDH’, db2);
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 28/28
Grazie dell’attenzione
E. Somma (SIA-BdI) WMDB 2008 21/01/2008 29/28