1 2

Trng i hc cng ngh

Nguyn h nam, nguyn tr thnh, h quang thy

Gio trnh khai ph d liu

Nh xut bn HQGHN H Ni - 2012

3 4

Mc lc

Li gii thiu1.1. Nhu cu pht hin tri thc t d liu

1.1.1. Tnh trng bng n d liu1.1.2. Ngnh cng nghip da trn d liu

1.2. Khi nim Pht hin tri thc trong c s d liu1.2.1. Gii thch mt s thut ng1.2.2. Qu trnh pht hin tri thc trong c s d liu1.2.3. Bc khai ph d liu trong qu trnh pht hin tri thc t d liu1.2.4. Kin trc mt h thng khai ph d liu

1.3. Khai ph d liu v x l CSDL truyn thng1.4. Mt s lnh vc ng dng khai ph d liu in hnh

1.5. Kiu d liu trong khai ph d liu1.5.1. C s d liu quan h1.5.2. Kho d liu1.5.3. C s d liu giao dch1.5.4. Cc h thng d liu m rng

1.6. Cc bi ton khai ph d liu in hnh1.6.1. M t khi nim1.6.2. Quan h kt hp1.6.3. Phn lp1.6.4. Phn cm1.6.5. Hi quy1.6.6. M hnh ph thuc1.6.7. Pht hin bin i v lch

1.7. Tnh lin ngnh ca khai ph d liuCu hi v Bi tp

2.1. Vai tr ca CNTT trong kinh t tri thc2.1.1. Nghch l hiu qu ca CNTT ca Robert Solow v lun im ca N. Carr2.1.2. Vai tr ca CNTT trong nn Kinh t tri thc2.1.2. Vai tr ca gim c thng tin trong doanh nghip v t chc

2.2. Cng ngh tri thc2.2.1. Khi nim tri thc2.2.2. Ngun tri thc cho c nhn v t chc2.2.3. Cng ngh tri thc

2.3. Bi ton pht hin tri thc t d liu2.3.1. S tin ha ca m hnh pht hin tri thc2.3.2 V bi ton khai ph d liu

2.4. o hp dn trong khai ph d liu2.4.1. Gii thiu chung v o hp dn

Cu hi v Bi tp3.1. Gii thiu3.2. Hiu d liu

3.2.1. o tp trung ca d liu3.2.2. o phn tn ca d liu3.2.3. Hin th d liu tm tt

3.3. Tin x l d liu3.4. Lm sch d liu

3.4.1. Cc gi tr b thiu3.4.2. D liu b nhiu3.4.3. Lm sch d liu phi l mt quy trnh

3.4. Tch hp d liu

5 6

3.4.1. Nhn din thc th3.4.2. S d tha v phn tch tng quan3.4.3. Pht hin cc b lp3.4.4. Pht hin xung t trong d liu v mc tru tng.

3.5. Chuyn i d liu3.5.1. Cc chin lc chuyn i d liu3.5.2. Chun ha d liu

3.6. Phng php thu gn d liu3.6.1. Gim chiu d liu3.6.2. Gim s lng d liu

3.7. Ri rc ha d liu v sinh cy khi nim phn cp3.7.1. Phng php p dng cho d liu s3.7.Phng php p dng cho d liu phn loi

3.8. Tng ktCu hi v Bi tp4.1 Gii thiu v lut kt hp4.2 Phng php khai ph tp mc ph bin

4.2.1. tng thut ton Apriori4.3. Thut ton FP-Growth4.3.1. tng thut ton4.3.2. Thut ton FP-growth.4.4 Mt s thut ton song song

4.4.1. Thut ton phn phi h tr4.4.2. Thut ton phn phi d liu

4.4.3. Thut ton phn phi tp ng c vin4.4.4. Thut ton sinh lut song song4.4.5. Mt s thut ton khc

4.5 Mt s ng dng ca lut kt hp

Cu hi v Bi tp5.1. Gii thiu

5.1.1. Bi ton phn cm5.1.2. Cc phng php phn cm

5.2. Mt s o c bn dng trong phn cm5.2.1. o tng ng5.2.2. o khc bit

5.3. Thut ton phn cm phng5.3.1. Thut ton k-means5.3.2 Thut ton k-mediods5.3.3. Tm s lng cm thch hp

5.4. Thut ton phn cm phn cp5.4.1 Phn cm phn cp gp (Hierarchical Agglomerative clustering HAC)5.4.2 Cc thut phn cm phn cp BIRCH5.4.3 Thut ton phn cm phn cp t trn xung DIANA5.4.4 Thut ton phn cm phn cp ROCK

5.5. Thut ton phn cm da trn mt 5.6. Gii thut phn cm da trn m hnh5.7. Nhn xt s b cc thut ton phn cm5.8. nh gi cc gii thut phn cm

5.8.1 nh gi da trn tng t5.8.2 nh gi da trn d liu gn nhn

5.9. Mt s ng dng ca phn cmCu hi v bi tp6.1. Gii thiu6.2. Phn lp bng cy quyt nh

7 8

6.2.1 li thng tin6.2.2 T s li6.2.3 Ch s Gini6.2.4 Ta cy quyt nh

6.3. Thut ton phn lp Naive Bayes6.2.1 nh l Bayes6.2.2 Phn lp Naive Bayes

6.3. Thut ton phn lp my vector h tr SVM6.3.1 Trng hp d liu c th phn loi tuyn tnh6.3.2 Trng hp d liu khng th phn tch tuyn tnh6.3.3 Phn lp a lp vi SVM

6.4. Thut ton phn lp kNN6.5. nh gi cc gii thut phn lp6.6. Mt s ng dng ca cc gii thut phn lpCu hi v bi tp7.1. Gii thiu7.2 Thut ton cc i k vng EM7.2. Thut ton hc cng tc

7.2.1. Thut ton hc cng tc da trn nhiu khung nhn7.2.2. Thut ton hc cng tc co-EM7.2.3. Thut ton hc cng tc da trn nhiu gii thut hc gim st

7.3. Thut ton Tri-training7.4 Thut ton t hun luyn Shelf-training7.5 Mt s ng dng ca cc gii thut hc bn gim stCu hi v bi tp8.1. Kha cnh php lut bo v tnh ring t v khai ph d liu

8.1.1. Hng dn ca OECD v d liu ring t v tc ng ti hot ng pht hin tri thc t d liu8.1.2. Tip cn php lut bo v tnh ring t ti nc M v tc ng ti khai ph d liu8.2. Phng php khai ph d liu bo v tnh ring t

8.2.1. M hnh v phng php khai ph d liu bo v tnh ring t8.2.2. Mt s thut ton khai ph d liu bo v tnh ring t

Cu hi v Bi tp9.1. Phng php tp m trong khai ph d liu

9.1.1. Mt s kin thc c s ca l thuyt tp m9.1.2. Phng php tp m trong khai ph d liu

9.2. Phng php tp th trong khai ph d liu9.2.1. Mt s kin thc c s v l thuyt tp th9.2.2. Phng php tp th rt gn thuc tnh9.2.3. Phng php tp th ri rc tp gi tr thuc tnh

9.3. Phng php tp m-th trong khai ph d liu9.3.1. La chn thuc tnh da trn tp m - th9.3.2. Phn lp k-NN da trn tp m - th

Cu hi v Bi tp10.1. Mt s bi hc trong khai ph d liu

10.1.1. Bi hc v k thut10.1.2. Bi hc v trin khai d n10.1.3. c trng ca chuyn vin khai ph d liu

10.2. Mt s li thng gp trong khai ph d liu10.3. Cng c Khai ph d liu

9 10

10.3.1. Tiu ch phn loi cc cng c khai ph d liu10.3.2. Cc kiu cng c khai ph d liu10.3.3. Tp v d nh gi cng c nghin cu

10.4. Khuynh hng pht trin ca khai ph d liu10.4.1. Khuynh hng pht trin ca khoa hc my tnh10.4.2. Khuynh hng pht trin ca khai ph d liu

Cu hi v Bi tpTi liu tham kho

11 12

Li gii thiu

Trong thi i ngy nay, s dng tri thc tr thnh ng lc ch cht cho tng trng kinh t quc gia, cho tng cng nng lc cnh tranh ca doanh nghip. ng thi, dung lng d liu s tng rt nhanh chng, c bit loi d liu do ngi s dng to ra (User-Generated Content: UGC) chim t trng ngy cng cao, tr thnh ngun ti nguyn tim n thng tin v tri thc c tim nng ln hu ch cho pht trin kinh t v tng cng nng lc cnh tranh. Nghin cu v trin khai cc phng php t ng pht hin cc mu mi, c gi tr, hu ch tim nng v hiu c trong khi d liu s, khc phc hin tng "giu v d liu m ngho v thng tin, hng ti mc tiu tng cng ti nguyn tri thc l ht sc cn thit v c ngha. Khai ph d liu (Data Mining) v Pht hin tri thc trong c s d liu (Knowledge Discovery in Data Bases: KDD), thnh phn quan trng ca cng ngh tri thc (Knowledge Technology), ang pht trin rt mnh m.

Khai ph d liu l mn hc bt buc trong chng trnh o to ngnh H thng thng tin (HTTT) bc c nhn v chuyn ngnh HTTT bc thc s ti Khoa CNTT, Trng i hc Cng ngh (HCN), i hc Quc gia H Ni (HQGHN). Nhu cu o to, nghin cu v pht trin lnh vc khai ph d liu trc ht ti Trng HCN, v sau ti cc c s o to v nghin cu trong nc i hi mt gio trnh c ni dung ton din v lnh vc nghin cu v trin khai quan trng ny.

Trc khi gii thiu ni dung ca gio trnh ny, chng ti mun nu ln mt vi im v cch tip cn ca chng ti. Th nht, gio trnh c vit phc v vic ging dy v hc tp bc i hc v bc sau i hc ti Trng HCN, HQGHN. Ni dung trong gio trnh c tng hp v tm lc t mt s ti liu ni ting cng nh nhng nghin cu thi s nht v khai ph d liu. Th hai, ni dung

v kho d liu c vit thnh gio trnh "Kho d liu" cho nn s khng c a vo gio trnh ny. Th ba, gio trnh ny cn c mc tiu nh hng cho cc nghin cu chuyn su v khai ph d liu, v vy, gio trnh b sung thm mt s ni dung khc vi nhiu cun sch hin c v khai ph d liu. Ni dung u tin c b sung l mt s kin thc v tri thc v kinh t tri thc. Thm na, chng ti b sung mt s ni dung v khai ph d liu da trn l thuyt tp m, l thuyt tp th v mt s bi hc thnh cng cng nh mt s li thng gp trong khai ph d liu. Khuynh hng nghin cu v trin khai khai ph d liu c trnh by vi cc ni dung cp nht nht c th c.

Gio trnh gm 10 chng vi ni dung s b nh c trnh by di y.

Chng 1. Gii thiu chung v khai ph d liu trnh by v s tng trng mnh m v dung lng d liu (c bit l d liu ni dung do ngi dng sinh ra: gerenated user content GUC), v cng ngh da trn d liu, v nhu cu pht hin tri thc t d liu, v cc khi nim c bn nht ca khai ph d liu v pht hin tri thc t d liu. Tnh lin ngnh ca khai ph d liu v s phn bit gia h thng khai ph d liu v h thng qun l c s d liu, gia bi ton khai ph d liu v bi ton thng k cng c cp.

Chng 2. Cng ngh tri thc v pht hin tri thc t d liu cung cp nhng kin thc c bn nht v tri thc v kinh t tri thc, vai tr ca CNTT v cng ngh tri thc cho pht trin kinh t v to li th cnh tranh. Qu trnh tin ha ca m hnh pht hin tri thc t d liu c phn tch. Mt s ni dung v o hp dn v tnh hp dn ca mu c trnh by.

Chng 3 Chun b d liu v kho d liu cung cp cc kin thc v k nng v hiu d liu, tin x l d liu, chuyn dng d liu, la chn thuc tnh.

13 14

Chng 4. Pht hin lut kt hp trnh by khi nim lut kt hp, mt s thut ton khai ph lut kt hp in hnh (thut ton Apriori, thut ton FP-growth v v mt s thut ton khc), khi nim lut dy v khai ph lut dy. Mt s ng dng ca lut kt hp cng c gii thiu.

Chng 5. Phn cm d liu v m t cung cp kin thc v bi ton phn cm v mt s thut ton phn cm in hnh (phn cm phn cp, phn cm phng K-mean, phn cm EM, mt s thut ton khc). Phng php nh gi phn cm v mt s ng dng phn cm cng c gii thiu.

Chng 6. Phn lp d liu trnh by v khi nim bi ton phn lp, mt s thut ton phn lp in hnh (C4.5, Naive Bayes, k-NN, SVM v mt s thut ton khc). Phng php nh gi thut ton phn lp v mt s ng dng thut ton phn lp cng c gii thiu.

Chng 7. Phng php hc bn gim st c bt u bng cc ni dung c bn ca phng php hc bn gim st. Mt s thut ton bn gim st in hnh (Adaboost, Co-training, Shelf-training v mt s thut ton hc bn gim st khc) c trnh by chi tit. Mt s ng dng hc bn gim st cng c gii thiu.

Chng 8. Khai ph d liu bo v tnh ring t cung cp cc kin thc c bn v tnh ring t, mt s m hnh v giI php khai ph d liu bo vi tnh ring t.

Chng 9. Tp m, tp th v tp m-th trong khai ph d liu trnh by mt s kin thc c bn v tp m, tp th, tp m-th v ng dng cc tp ni trn trong khai ph d liu.

Chng 10. Mt s bi hc v khuynh hng pht trin ca khai ph d liu trnh by mt s bi hc v li thng gp trong khai ph d liu. Phn cui ca chng cp ti khung hng pht trin khai ph d liu, tp trung vo, khai ph d liu phng tin x hi, hc

my hng min ng dngv hc my khng dng c chn la gii thiu chi tit hn.

Gio trnh ny c s dng cho c bc i hc v bc cao hc. Mt phng n ngh cho o to bc i hc l gi ni dung bao gm chng 1, chng 2 (khng k mc 2.4), chng 3 (khng k mc 3.4), chng 4, chng 5, chng 6, chng 10 (hai mc 10.1, 10.2). n li ni dung dnh cho bc i hc v nghin cu cc ni dung cn li trong gio trnh l phng n ni dung dy-hc cho bc sau i hc.

i vi lnh vc khai ph d liu, vic dng thut ng ting Vit l rt kh khn v y l lnh vc nghin cu cn rt mi khng ch Vit Nam m cn trn th gii. Vi mi thut ng ting Anh, thut ng ting Vit tng ng c coi l ph bin c chn la.

Nhm tc gi xin by t li cm n chn thnh ti TS. Nguyn L Minh, TS. on Sn, TS. Phan Xun Hiu, TS. Nguyn Cm T, TS. Nguyn Vit Cng, TS. ng Thanh Hi nhit tnh cng tc. Nhm tc gi nh gi cao v chn thnh cm n tp th cn b, sinh vin thuc Phng Th nghim Cng ngh tri thc v B mn HTTT, Khoa CNTT. Gio trnh ny cng l mt sn phm ca qu trnh cng tc nghin cu ca chng ti vi C Gio s Susumu Horiguchi ti Vin Khoa hc & Cng ngh tin tin v i hc Tohoku Nht Bn, GS. Akira Shimazu ti Vin Khoa hc & Cng ngh tin tin Nht Bn, TSKH Nguyn Hng Sn ti i hc Vasava Ba Lan.

D nhm tc gi c gng thu thp, nghin cu v tng hp song gio trnh chc chn cn khng t khim khuyt. Chng ti mong mun nhn c s cm thng cng nh cc kin ng gp t cc nh khoa hc, cc ging vin v ngi hc gio trnh ngy cng thm hon thin.

Nhm tc gi xin chn thnh cm n cc c quan hu quan tch cc h tr xut bn gio trnh.

15 16

Chng 1. Gii thiu chung v khai ph d liu

Chng m u ca gio trnh trnh by mt s nt khi qut nht v khai ph d liu. Mc u tin gii thiu v tnh t nhin ca tnh trng bng n d liu v pht hin tri thc t d liu nh mt thnh phn nn tng cng ngh ca ngnh kinh t nh hng d liu. Mc th hai gii thiu khi nim pht hin tri thc trong c s d liu, khi nim khai ph d liu. Pht hin tri thc trong c s d liu l mt qu trnh gm nhiu bc tm ra nhng mu c gi tr, mi, hu ch tim nng v hiu c trong mt tp d liu ln. Khai ph d liu l bc x l c th nht ca qu trnh ny, v vy, trong khng t trng hp hai khi nim pht hin tri thc t d liu v khai ph d liu c dng thay th nhau. Mc tip theo trnh by mt s kha cnh phn bit h thng khai ph d liu (cung cp thng tin h tr quyt nh) vi h thng c s d liu iu hnh tc nghip truyn thng (phc v x l giao dch tc nghip). Mc th t gii thiu mt s lnh vc ng dng khai ph in hnh, trong kinh doanh l mt trong nhng lnh vc ng dng ph bin nht. Mc th nm cung cp mt s thng tin cho bit tnh a dng ca kiu d liu u vo ca bi ton khai ph d liu. Mc th su gii thiu cc bi ton khai ph d liu in hnh thuc vo hai lp bi ton d bo v m t. Mc cui cng ca chng ny trnh by tnh a ngnh ca lnh vc khai ph d liu.

1.1. Nhu cu pht hin tri thc t d liu

1.1.1. Tnh trng bng n d liu

Thi i ngy nay, mi ngi chng ta tng nghe ni v chng kin v s tng trng lin tc vi tc vt bc v dung lng d liu do con ngi khi to, lu gi v truyn dn; s tng trng ny cn c gi l hin tng bng n thng tin. Trc khi xem xt mi lin h gia hin tng bng n thng tin vi nhu cu khai ph d liu v pht hin tri thc t d liu, chng ta tm hiu v cc nguyn nhn to nn hin tng bng n thng tin . Ni mt cch khi qut, hin tng bng n thng tin c nguyn nhn t nhu cu hot ng mi mt ca i sng x hi, tuy nhin, nhng ni dung trnh by di y s lm chi tit hn v cc kha cnh cng ngh v x hi gp phn thc y s tng trng d liu vt bc .

Hnh 1.1. Xu th ti u chi ph sn xut mch bn dn: S lng

thnh phn bn dn trong mt mch tch hp tng v chi ph sn xut mt thnh phn bn dn gim (G.E. Moore, 1965 [Moore65]).

17 18

1.1.1.1. V mt cng ngh

Bng 1.1. Tng giao vn IP nm 2009 v d bo cc nm 2010-2014. Ch thch: Consumer: Lu lng IP c nh do h gia nh, c dn trng i hc, v c ph Internet to ra; Business: Lu lng IP hoc WAN c nh (khng bao gm lu lng sao lu) do doanh nghip v chnh quyn to ra; Mobility: Lu lng d liu di ng v truy cp Internet t thit b cm tay, th my tnh xch tay, WiMAX; Internet: ton b lu lng IP i qua ng trc Internet. Ngun: Sch trng CISCO 2010

S tng trng d liu vi tc cao nh c cp c dn xut t cc nguyn nhn cng ngh sau y:

+ Cng ngh ch to cc thit b x l, lu gi v truyn dn d liu v ang pht trin khng ngng, to ra cc sn phm thit b c tc hot ng ngy cng cao v gi thnh ngy cng h. S pht trin cng ngh ny c dn dt bi nh lut Moore, mt nh lut c xut pht im t ni dung mt bi bo c Gordon E. Moore, mt ng sng lp cng ty Intel (INTegrated ELectronics) cng b vo nm 1965 [Moore65]. Ni dung c coi l quan trng nht trong bi bo ny ca G. E. Moore l d bo v xu th tng s lng thnh phn bn dn t c chi ph sn xut hiu qu nht (Hnh 1.1).

Sau ny, d bo ni trn ca G.E. Moore c pht biu di dng phng ngn 2x nh sau S lng bn dn tch hp trong mt chp s tng gp i sau mt chu k khong hai nm. Mt dng pht biu khc ca nh lut Moore l chi ph sn xut mch bn dn vi cng tnh nng gim mt na sau khong hai nm. Phin bn 18 thng ca phng ngn 2x rt ngn chu k thi gian t hai nm xung cn 18 thng.

nh lut Moore v ang dn dt ngnh cng nghip mch bn dn m v bn cht, n l m hnh c bn cho ngnh cng nghip bn dn. Theo Paul S. Otellini, Ch tch v Gim c iu hnh Tp on Intel, th nh lut Moore vn to kh nng c bn cho s pht trin ca chng ti, v n vn cn hiu lc tt ti Intel. Nhng cch chng ti v khch hng xem xt nh lut Moore c s thay i. nh lut Moore khng ch l mch bn dn. N cng l cch s dng sng to mch bn dn1. Theo Daniel Grupp, Gim c pht trin cng ngh tin tin ca Acorn Technologies, Inc. (http://acorntech.com/) th ton b chu trnh thit k, pht trin, sn xut, phn phi v bn hng c coi l c tnh bn vng khi tun theo nh lut Moore.. Nu nh bi nh lut Moore, th trng khng th hp th ht cc sn phm mi, v k s b mt vic lm. Nu b tt sau nh lut Moore, khng c g mua, v gnh nng ln i vai ca chui nh phn phi sn phm2.

Cuc cch mng trong cng nghip mch bn dn (nn tng ca cng nghip in t) tc ng mnh m i vi cng nghip phn cng my tnh, to ra s bng n v nng lc x l tnh ton v dung lng lu tr d liu; kt qu l cc thit b to lp v lu tr d liu

1 Intel Silicon Innovation. http://download.intel.com/museum/Moores_Law/ Printed_Materials/Intel_Silicon_Brochure.pdf 2 http://www.edavision.com/200111/feature.pdf

19 20

mang theo s tin b cng ngh khng ngng c sn xut v a vo s dng.

Lich s pht trin cc b x l Intel l mt minh ha in hnh, th hin s pht trin cng ngh b x l c dn dt bi nh lut Moore3. Mt v d khc, hot ng thu thp d liu ca Sloan Digital Sky Survey (SDSS) - t chc hp tc quc t ln nht v kho st thin vn bt u lm vic t nm 2000 l mt minh chng in hnh v s pht trin ca cng ngh thu thp d liu. Trong vi tun hot ng u tin, knh vin vng u tin ca SDSS ti New Mexico thu thp c lng d liu nhiu hn dung lng d liu c tch ly trong ton b lch s thin vn hc trc . Hin ti, sau mt thp k, kho ti nguyn d liu ca SDSS ln ti 140 TB. Knh vin vng k tip ca SDSS (Large Synoptic Survey Telescope) t ti Chile, c bt u hot ng vo nm 2016, s thu nhn c khi lng d liu nh vy (140 TB) ch trong nm ngy.

Cc kt qu ca s pht trin cng ngh phn cng my tnh to iu kin thun li cho s pht trin cng ngh c s d liu (lin quan ti hot ng t chc v qun l d liu) v cng ngh mng (lin quan ti hot ng truyn dn d liu), hp thnh mt nn tng k thut tng hp cho s bng n thng tin.

+ Cng ngh CSDL v ang pht trin khng ngng nhm p ng nhu cu qun l d liu ngy cng nng cao ca x hi loi ngi (ni chung) v trong hot ng qun l (ni ring). Hnh 1.2 trnh by qu trnh tin ha cng ngh CSDL theo quan im ca J. Han v M. Kamber [HK0106].

Trong qu trnh tin ha ca cng ngh CSDL, nhiu h qun tr c s d liu c pht trin v nng lc ca h qun tr c s d liu cng ngy c nng cao. S tng trng ni bt v kch thc ca c s d liu qun l dn n c nhiu c s d liu vi kch thc

3 Intel 40th Anniversary backgrounder. http://www.intel.com/pressroom/enhanced/40th_Anniversary/ 40th_anniversary_backgrounder.pdf?iid=pr_smrelease_40th_addlmat1

Hnh 1.2. Tin ha ca cng ngh c s d liu [HK0106]

21 22

hng trm TB (1TB = 1000 GB) xut hin. Chng hn, c s d liu ca Trung tm tnh ton khoa hc nghin cu nng lng quc gia M (National Energy Research Scientific Computing Center: NERSC) ti thng 3/2010 t khong 460 TB4. C s d liu ca YouTube sau hai nm hot ng c ti hng trm triu video, dung lng c s d liu ca YouTube tng gp i sau mi chu k 5 thng. H thng siu th bn l Wal-Mart, mi gi c hn 1 M giao dch khch hng, cung cp cc c s d liu m dung lng chung c tnh ln ti hn 2,5 PB (1 PB = 1000 TB5).

+ S pht trin cng ngh mng c v quy m v tc to ra s tng trng mnh m v nng lc truyn dn thng tin. Theo bo co tng hp ca CISCO, tng dung lng d liu thng qua giao vn IP trong mt thng tng t 14.686 PB vo nm 2009 ln 20.396 PB vo nm 2010 v d bo ln ti 63.463 PB vo nm 2014. Theo d bo, tng trung bnh hng nm v dung lng d liu qua giao vn IP trong giai on 2009-2014 t khong 34% (Bng 1.1).

c bit, World Wide Web tr thnh mng thng tin khng l, trong s lng trang Web c nh ch s ln ti con s hng chc t (theo s liu cng b vo ngy 23/01/2011 ca WorldWideWeb.com, c hn 13 t ri trang Web c nh ch s)6.

4 http://www.nersc.gov/news/annual_reports/annrep0809/annrep0809.pdf 5 Dy n v o dung lng nh c xp theo chiu tng 1000 ln: Byte (B), Kilo bytes (KB), Mega B (MB), Giga B (GB), Texa B (TB), Peta B (PB), Exa B (EB), Zetta B (ZB), Yotta B (YB). Nh vy, 1 EB = 1 t GB v 1 ZB = 1 nghn t GB. 6 http://www.worldwidewebsize.com/

Hnh 1.3. Dung lng d liu tng th nm 2010 t khong 1.260 EB (1EB = 1t GB) [IDC10]. Ch thch: (1) Ngi dng v nhn vin to, lu gi, hoc sao chp thng tin c nhn; (2) Doanh nghip to, vn chuyn, lu tr, qun l, hoc bo mt.

1.1.1.2. V mt x hi

Xu th pht trin x hi thng tin m rng i ng tc nhn to lp v s dng d liu. Ngun d liu c to lp, khai thc v truyn dn khng ch c trong hot ng tc nghip ti cc cng ty, m mt lng d liu khng l khc c mt lc lng hng hu cc c nhn to lp v ph bin trn Internet trn cc trang web c nhn, cc mng x hi... Ti thng 2/2011, mng x hi Facebook bao gm khong 40 t nh7.

Ti Hnh 1.3, vo nm 2010, dung lng d liu tng th ton th gii t khong 1.260 EB, trong c ti 900 EB d liu do ngi s dng to ra (UGC: User-Genegated Content); dung lng d

7 http://www.economist.com/node/15557443?story_id=15557443, ng ngy 25/2/2010.

23 24

liu loi ny gp gn 4 ln dung lng d liu c cc doanh nghip to lp ra (khong 240 EB).

1.1.1.3. Chi ph to lp d liu mi ngy cng gim

Theo tnh ton d bo ca IDC c cng b vo thng 5/2010, gi thnh to mi 1 GB d liu l gn 0,5 xu M vo nm 2009; gi thnh ny s tip tc gim trong cc nm tip theo v d kin gi to mi mt GB d liu s vo khong 0,02 xu M vo nm 2020 (Hnh 1.4). iu c li ny va l kt qu ca cuc cch mng cng ngh va l mt nguyn nhn gp phn tng trng dung lng d liu.

Hnh 1.4. Dung lng d liu tng th v gi thnh to lp d liu giai

on 2009-2020 [IDC10].

Sau y l mt s v d minh ha v tnh phong ph ca hin tng bng n d liu. D liu tng th tip tc pht sinh, lu tr bao gm giao dch thng mi, cuc gi din thoi, d liu khoa hc: thin vn, sinh hc, Web, vn bn, nh,... Theo tng hp ca IDC, tuy c b nh hng ca khng hong kinh t trong cc nm 2008-2009 song dung lng d liu tng th vo nm 2010 tng 62% so vi gn 0,8 ZB (800 EB) vo nm 2009 t ti 1,26 ZB (1260 EB).

Cng theo IDC, ngun d liu tng th c d bo ln ti 35 ZB vo nm 2020. dc ca ng biu din dung lng ngun d liu

tng th trong Hnh 1.4 ngy cng ln, chng t tng trng d liu ngy cng cao.

1.1.2. Ngnh cng nghip da trn d liu

Vic to lp, thu thp v lu tr d liu vi kt qu l xut hin cc kho cha d liu khng l c lit k trn y khng ngoi mc ch khai ph d liu nhm pht hin cc tri thc mi gip ch cho hot ng ca con ngi trong tp hp d liu.

Theo Jim Gray, chuyn gia ca Microsoft, ngi c nhn gii thng Turing nm 1998, th Chng ta ang ngp trong d liu khoa hc, d liu y t, d liu nhn khu hc, d liu ti chnh, v cc d liu tip th. Con ngi khng c thi gian xem xt d liu nh vy V vy, chng ta phi tm cch t ng phn tch d liu, t ng phn loi n, t ng tm tt n, t ng pht hin v m t cc xu hng trong n, v t ng ch dn cc d thng. y l mt trong nhng lnh vc nng ng v th v nht ca cng ng nghin cu c s d liu. Cc nh nghin cu v thng k, trc quan ha, tr tu nhn to, v hc my ang ng gp cho lnh vc ny. Tnh rng ln ca lnh vc lm cho n tr nn kh khn nm bt nhng tin b phi thng trong vi thp k gn y [HK0106].

Kenneth Cukier a ra nhn nh tng t Thng tin t khan him ti d dt. iu mang li li ch mi to ln... to nn kh nng lm c nhiu vic m trc y khng th thc hin c: nhn ra cc xu hng kinh doanh, ngn nga bnh tt, chng ti phm...c qun l tt, d liu nh vy c th c s dng m kha cc ngun mi c gi tr kinh t, cung cp nhng hiu bit mi vo khoa hc v to ra li ch t qun l..."7.

Nh c trnh by, nhiu tri thc c ch ang tim n trong tp d liu s c thu thp v lu gi. Tuy nhin, dung lng khng l ca d liu c to lp, thu thp v lu tr li to nn cc

25 26

thch thc mi cho con ngi trong vic hiu v x l d liu, dn n tnh trng con ngi ngp trong d liu. Cng theo Kenneth Cukier7 th thng tin t khan him ti d dt cng l mt ni au u ln... Con ngi t lu phn nn rng h phi bi trong thng tin (d liu). Ni au u m Kenneth Cukier mun ni n l hin tng con ngi ngy cng kh tip cn c ngun d liu to ln ang ngy cng gia tng cng nh kh tip cn c cch thc qun l tt c ngun d liu khng l nhm mang li li ch to ln trong vic nhn ra cc xu hng kinh doanh, ngn nga bnh tt, chng ti phm

Hnh 1.5 cung cp mt d bo IDC Digital Universe Study v tng d liu tng th trong v tr s giai on 2009-2020. So vi nm 2009, vo nm 2020, s lng i tng cha tin tng 67 ln, dung lng d liu c t 35 ZB tng 44 ln, tuy nhin lc lng lao ng v CNTT ch tng 1,4 ln. S chnh lch gia tc tng dung lng d liu so vi tc tng lc lng lao ng CNTT cng tr thnh mt thch thc ln trong x l v s dng d liu.

Hn na, th gii ngy nay ang trong thi k qu chuyn t kinh t hng ha (good economy, hay kinh t hng hng ha: good-dominant economy) sang kinh t dch v (service economy, hay kinh t hng dch v: service-dominant economy), m tri thc tr thnh ng lc ch cht cho tng trng kinh t (Chng 2). Trong xu th chuyn sang nn kinh t hng dch v, yu cu qun l tt d liu ln cng tr nn cp bch i vi con ngi trong vic gii quyt tnh trng ngp trong d liu m kht tri thc.

Nh vy, mt yu cu cp thit t ra l phi xy dng c cc phng php mi x l (t ng) d liu ph hp vi hon cnh khi lng d liu rt ln v ang tng trng vi tc ngy cng cao. Cc nh nghin cu v trin khai cp ti cuc cch mng cng nghip d liu (the industrial revolution of data). Mt lnh vc khoa hc mi mang tn khoa hc d liu ln7 (science of big

data) c hnh thnh. T ngun d liu khng l c qun l tt, chng ta s thu nhn c cc tri thc v xu hng kinh doanh, v ngn nga bnh tt, v chng ti phm.

Hnh 1.5. tng ca d liu tng th v lc lng lao ng CNTT

giai on 2009-2020 [IDC10].

Cng theo Kenneth Cukier, cng nghip qun l v phn tch d liu nhn c tri thc tim n t d liu (cng nghip da trn d liu) c nh gi ln ti hn 100 t la M ti thi im nm 2010 v c tc tng trng khong 10% hng nm (gn gp i so vi tc tng trng ca kinh doanh phn mm ni chung). Trong mt vi nm cui ca thp nin 2000, cc tp on CNTT hng u th gii nh Oracle, IBM, Microsoft v SAP chi ti hn 15 t la M mua li cc cng ty phn mm chuyn v qun l v phn tch d liu.

nh ngha cng ngh d liu ln ca IDC vo nm 2011 [GR11] cung cp mt cch hiu v ni dung ca cng ngh mi ny: Cng ngh d liu ln m t mt th h mi ca cng ngh v kin trc h tng, c thit k tit kim nht thu c gi tr t khi lng

27 28

rt ln ca d liu a dng, bng cch cho php chp tc cao, pht hin v/hoc phn tch8.

Song hnh vi xu hng hot ng qun l, pht hin v phn tch d liu ngy cng c tng cng, ngi gim c thng tin (Chief information officer: CIO) c vai tr ngy cng ni bt trong b my iu hnh ca t chc. H l cc nh khoa hc d liu (data scientist), nhng ngi tch hp c cc k nng ca lp trnh vin, nh thng k v ngh nhn nhm o c vng cm n trong ni d liu. c im ngh nhn ca nh khoa hc d liu cn c ch dn nh l ngi k chuyn (storyteller). iu ny c th c gii thch l nh khoa hc d liu c nng lc k li c cu chuyn ca d liu, t cho php nhn ra c cc tri thc hu ch, cn thit t ni d liu s. Theo Qu khoa hc quc gia M (NSF), nh khoa hc d liu c cc chc nng sau y thi hnh sng to hot ng kho st v phn tch, tng cng t vn, hp tc, v phi hp nng lc ca nhng ngi khc tin hnh nghin cu v gio dc bng cc b d liu s; i tin phong trong vic pht trin sng to trong lnh vc cng ngh c s d liu v khoa hc thng tin, bao gm phng php trc quan ha d liu v pht hin tri thc p dng vo cc lnh vc khoa hc v gio dc lin quan n cc b d liu; thi hnh mt cch tt nht c theo kha cnh thc tin ln kha cnh cng ngh; ng vai tr c vn khi to hoc chuyn i d liu cho cc nh iu tra, sinh vin v nhng ngi khc c quan tm ti khoa hc d liu; thit k v thi hnh cc chng trnh gio dc v tip cn cng ng lm cho li ch ca cc b d liu v thng tin khoa hc k thut s ti cc nghin cu vin, ging vin, sinh vin v cng chng trong mt phm vi rng nht c th c [NSF05]. A.

8 Nguyn vn: "Big data technologies describe a new generation of technologies and architectures, designed to economically extract value from very large volumes of a wide variety of data, by enabling high-velocity capture, discovery, and/or analysis".

Swan v S. Brown [SB08] quan nim rng nh khoa hc d liu l nhng ngi nghin cu v thc hin ton b hoc b phn tp hp cc chc nng nh nh ngha trn y ca NSF. Hai tc gi phn bit nh khoa hc d liu vi nh to lp d liu (tc gi d liu), ngi qun l d liu, v chuyn vin th vin. Mt nh khoa hc d liu hoc l nh khoa hc my tnh c k nng ng k min ng dng hoc l nh khoa hc min ng dng c k nng ng k v tnh ton.

Thch ng vi hon cnh d liu ln, h tr c lc cho nh khoa hc d liu l cc phng php x l d liu mi v cc b cng c tin ch thi hnh vi cc phng php ny pht hin ra cc tri thc mi, c gi tr, hu dng ang tim n trong d liu ln . Xy dng v pht trin phng php v cng c x l d liu ln nhm mc ch pht hin tri thc tim n l ni dung ca lnh vc pht hin tri thc trong c s d liu (Knowledge Discovery in Databases: KDD). Khai ph d liu (Data Mining) l bi ton x l d liu c bn nht trong qu trnh pht hin tri thc trong c s d liu. Trong nhiu trng hp, hai khi nim Khai ph d liu v Pht hin tri thc trong c s d liu cn mang cng mt ni dung.

Nh c m t trong Hnh 1.2, J. Han v M. Kamber [HK0106] cho rng qu trnh tin ha ca lnh vc cng ngh c s d liu (CSDL), trong cng ngh khai ph d liu (Data Mining) c coi l giai on tin ha mi ca cng ngh CSDL. Qu trnh tin ha ny c bt u t cui nhng nm 1980 v khng ngng c pht trin v b rng v chiu su.

29 30

1.2. Khi nim Pht hin tri thc trong c s d liu

Lnh vc khai ph d liu v pht hin tri thc trong CSDL l mt lnh vc rng ln, cun ht cc phng php, thut ton v k thut t nhiu chuyn ngnh nghin cu khc nhau nh hc my, thu nhn mu, CSDL, thng k, tr tu nhn to, thu nhn tri thc trong h chuyn gia cng hng ti mt mc tiu thng nht l trch lc ra c cc "tri thc" t d liu trong cc kho cha khng l. Tnh phong ph v a dng ca lnh vc khai ph d liu dn n mt thc trng l tn ti cc quan nim khc nhau v cc chuyn ngnh khoa hc - cng ngh gn gi nht vi lnh vc .

Hnh 1.6. Ngng n T theo thu nhp phn lp cho vay: Vng b ph bi ngng T (vng cc du chm) tng ng quyt nh khng

cho vay (FPS96] (Lu , vng pha trn ng nghing ri nt cho quyt nh tt hn).

Gio trnh ny tn thnh quan nim ca J. Han v M. Kamber coi lnh vc khai ph d liu l giai on pht trin mi ca cng ngh CSDL v c lin quan mt thit vi nhiu ngnh. Nh vy, c th gn lnh vc ny vi chuyn ngnh h thng thng tin.

V d 1.1. (Frawley, Piatetski-Shapiro v Matheus [FPS96])

Hnh 1.6. trnh by mt tp d liu gi nh v vay n ngn hng gm 23 trng hp c biu din trong khng gian hai chiu. Mi im trn th biu din mt trng hp vay n ngn hng trong qu kh. Trc honh biu din thu nhp cn trc tung biu din tng n c nhn ca ngi i vay (tin th chp, tin chi tr t...). D liu c phn thnh hai lp: lp gm nhng ngi thiu kh nng tr n ngn hng v lp o gm nhng ngi c tnh trng tt.

Hnh 1.7. Qu trnh pht hin tri thc trong c s d liu [FPS96]

T

Cho vay

Khng cho vay N

Thu nhp

31 32

Khi nim 1.1. [FPS96]

Pht hin tri thc trong c s d liu (i khi cn c gi l khai ph d liu) l mt qu trnh khng tm thng tm ra nhng mu c gi tr, mi, hu ch tim nng v hiu c trong d liu.

L lnh vc nghin cu v trin khai c pht trin rt nhanh chng v c phm vi rt rng ln, li c rt nhiu nhm nghin cu ti nhiu trng i hc, vin nghin cu, cng ty nhiu quc gia trn th gii quan tm, cho nn tn ti rt nhiu cch tip cn khc nhau i vi lnh vc pht hin tri thc trong CSDL. Chnh v l do m trong nhiu ti liu, nh ni trn, cc nh khoa hc trn th gii dng nhiu thut ng khc nhau, m cc thut ng ny c coi l mang cng ngha vi KDD nh chit lc tri thc (knowledge extraction), pht hin thng tin (information discovery), thu hoch thng tin (information harvesting), khai qut d liu (data archaeology), x l mu d liu (data pattern processing)... Hn na, trong nhiu trng hp, hai khi nim "Pht hin tri thc trong c s d liu" v "khai ph d liu" cn c dng thay th nhau [FPS96]. Hai khi nim khai ph d liu v pht hin tri thc trong cc CSDL thng cp i vi nhau.

1.2.1. Gii thch mt s thut ng

Mt s thut ng c trong nh ngha 1.1 trn y cn c gii thch l "mu", "c gi tr", "mi", "hu ch" v "hiu c". Di y trnh by mt s gii thch s b v cc khi nim ny nhm lm tng minh thm ng ngha ca khi nim KDD trong nh ngha 1.1.

D liu (chnh xc hn l tp d liu) c hiu nh l mt tp F gm hu hn cc trng hp (s kin). Theo ni dung ca pht hin tri thc trong cc CSDL, d liu phi bao gm nhiu trng hp. Trong v d 1.1, F l tp hp gm 23 trng hp (bn ghi) vi 3

trng thng tin (thuc tnh) tng ng cha cc gi tr v s n, thu nhp v tnh trng vay n. Trong bi ton khai ph vn bn, tp d liu F chnh tp hp cc vn bn c th c trong min ng dng. Trong bi ton khai ph lut kt hp giao dch, tp F bao gm tt c cc giao dch c th c c xem xt trong min p dng ca bi ton.

Mu: Trong qu trnh KDD, ngi ta s dng mt ngn ng L biu din cc tp con cc s kin (d liu) thuc vo tp s kin F, theo mi biu thc E trong ngn ng L s biu din mt tp con FE tng ng cc s kin trong F. E c gi l mu nu n n gin hn (theo mt ng cnh no ) so vi vic lit k cc s kin thuc FE. Chng hn, biu thc "THUNHP < $t" (m hnh cha mt bin THUNHP) trong mnh "Nu THUNHP < $t th ngi vay n ri vo tnh trng khng th chi tr" s l mt mu khi cho bin t nhn mt gi tr thch hp. Nh trnh by bng th ti Hnh 1.6, khi bin t nhn mt gi tr c th T mu ny (biu din mi trng hp c THUNHP < T) hin nhin l gn hn so vi vic lit k 14 trng hp c th. Tng t, nu F l tp cc trang Web trong kho lu tr ca mt my tm kim (chng hn Google) th mu "ti liu c cha t cm t "Search Engine" s biu din mt tp bao gm mt s lng rt ln cc ti liu Web c cha cm t "Search Engine" .

Qu trnh KDD thng bao gm nhiu bc l chun b d liu, tm kim mu, c lng tri thc, tinh ch s tng tc ni ti sau khi chuyn dng d liu. Qu trnh c tha nhn l khng tm thng theo ngha l qu trnh khng ch nhiu bc m cn c thc hin lp i lp li, v quan trng hn, qu trnh bao hm mt mc tm kim t ng. Chng hn trong V d 1.1, khi tnh ton ngha v thu nhp ca mt ngi, nu ch thng qua cc tc ng n gin m chng ta thu nhn c mt kt lun no c th l hu ch v mi quan h gia thu nhp v tnh trang vay ngn hng, chng hn nh ngi c thu nhp cao th kh nng th c tnh trng vay n

33 34

tt, th ng vi cho rng l mt khm ph (hoc ng cho rng mt tri thc c pht hin).

C gi tr: Mu c pht hin cn phi c gi tr i vi cc d liu mi (xut hin trong tng lai) theo mt mc chn thc no y. Tnh cht "c gi tr" c hiu theo ngha lin quan ti mt o tnh c gi tr (chn thc) l mt hm C nh x mt biu thc thuc ngn ng biu din mu L ti mt khng gian o c (b phn hoc ton b) MC. Mt biu thc E trong L biu din mt tp con FE F c th c gn mt o chn thc c = C(E,F).

Chng hn, nu ng bin xc nh mu "THUNHP < $t" nh ch dn trong Hnh 1.6 c dch sang phi (bin THUNHP nhn gi tr ln hn) th chn thc ca mu mi s b gim xung bi v n bao gi thm cc tnh hung vay tt li b a vo vng khng cho vay n.

Tng t, mu "Nu a*THUNHP + b*N < 0 (thuc m hnh tuyn tnh hai bin THUNHP v N trong a*THUNHP + b*N) th ngi vay n ri vo tnh trng khng th chi tr" biu din mt na mt phng pha trn ca ng ri nt trong Hnh 1.6 s cho chn thc cao hn (hay c coi l "c gi tr hn") so vi mi mu thuc m hnh mt bin "THUNHP < $t".

Tnh mi: Mu phi l mi trong mt min xem xt no , t nht l h thng ang c xem xt. Tnh mi c th o c khi quan tm ti s thay i trong d liu (bng vic so snh gi tr hin ti vi gi tr qu kh hoc gi tr k vng) hoc tri thc (tri thc mi quan h nh th no vi cc tri thc c). Tng qut, iu ny c th c o bng mt hm N(E,F) hoc l o v tnh mi hoc l o k vng.

Hu ch tim nng: Mu cn c kh nng ch dn ti cc tc ng hu dng v c o bi mt hm tin ch. Chng hn, hm U nh x cc biu thc trong L ti mt khng gian o c th t (b phn

hoc ton b) MU, theo u = U (E,F). V d, trong tp d liu vay n, hm ny c th l s tng hy vng theo s tng li ca nh bng (tnh theo n v tin t) kt hp vi quy tc quyt nh c trnh by trong Hnh 1.6.

C th hiu c: Mt mc tiu ca KDD l to ra cc mu m con ngi hiu chng d dng hn cc d liu nn (d liu sn c trong h thng). Chnh v l do tiu ch ny l kh m o c mt cch chnh xc cho nn thng tnh cht "c th hiu c" c thay bng mt o v s d hiu. Tn ti mt s o v s d hiu, cc o nh vy c sp xp t c php (tc l c ca mu theo bit) ti ng ngha (tc l d dng con ngi nhn thc c theo mt tc ng no ). Bi l do , chng ta gi nh rng tnh hiu c l o c bng mt hm S nh x biu thc E trong L ti mt khng gian o c c th t (b phn hoc ton b) MS; theo , s = S(E,F).

hp dn: Mt tiu ch quan trng, c gi l hp dn (interestingness), thng c coi nh mt o tng th v mu l s kt hp ca cc tiu ch gi tr, mi, hu ch v c th hiu c. Mt s h thng KDD thng s dng mt hm hp dn di dng hin i = I (E, F, C, N, U, S) thc hin nh x mt biu thc trong L vo mt khng gian o c Mi. Mt s h thng KDD khc li c th xc nh gi tr hp dn ca mu mt cch trc tip thng qua th t ca cc mu c pht hin.

Trong thc tin gii quyt cc bi ton khai ph d liu, ngi ta thng ch quan tm n hp dn, cn cc o khc c mc nh coi l thnh phn ca hp dn. C th l, khi thi hnh mt loi bi ton pht hin tri thc c th, mt s o tng ng c tnh ton nhm xc nh hp dn ca tri thc ("mu", "lut") ang c xem xt. Chng hn, trong bi ton khai ph lut kt hp, hai o c xem xt, l h tr (xc nh phm vi nh hng ca lut) v tin cy (xc nh tnh tin cy ca lut) hp thnh hp dn ca lut kt hp c khai ph. Tng t, trong bi ton phn

35 36

lp, ngi ta s dng hai o c bn l hi phc (kh nng bao gi v d ng) v chnh xc (kh nng chnh xc khi xc nh v d ng); ng thi, mt s o mang ngha kt hp t hai o ny cng c s dng.

Tri thc: Mt mu E L c gi l tri thc nu nh i vi mt lp ngi s dng no , ch ra c mt ngng i Mi m hp dn I(E,F,C,N,U,S) > i.

Ch rng nh ngha trn y v khi nim "tri thc" khng mang mt ngha tuyt i m ph thuc vo quan im ca ngi s dng h thng KDD ("mt lp ngi s dng no "). Nh mt ni dung ca s kin, n ch l mt nh hng cho ngi s dng v c xc nh bng bt k hm v ngng no c ngi s dng chn. Chng hn, trong bi ton khai ph lut kt hp, chng ta ch quan tm ti cc "tp ph bin" l nhng tp c h tr vt qua mt ngng minsup no . Hn na, ch cc lut kt hp c tin cy vt qu ngng minconf mi c khai ph cung cp tri thc ti ngi s dng. Cc ngng minsup v minconf c th c thay i theo la chn ca ngi s dng.

Theo cch hnh thc ha, thuyt minh chnh xc cho nh ngha trn y v "tri thc" l chn ngng no c MC (v tnh "c gi tr"), s MS (v tnh "c th hiu c") v u MU (v tnh "hu ch") v khi gi mu E l tri thc nu v ch nu:

C(E,F) > c v S(E,F) > s v U(E,F) >u

Thng qua vic t cc ngng thch hp vi mc ch pht hin tri thc, ngi s dng c th nhn mnh mt d bo chnh xc hoc cc mu hu ch (vt qua mt ngng o nh gi no ) qua nhng o lin quan. R rng l tn ti mt khng gian v hn cho php nh x I xc nh "tri thc cn pht hin". Quyt nh nh vy l t do i vi ngi s dng v c c trng i vi tng min ng dng.

Nghin cu v tnh hp dn ca mu v tri thc (c gi l o hp dn: interestingness measures) l mt ni dung nghin cu quan trng trong khai ph d liu v pht hin tri thc t d liu. Nhiu cng trnh nghin cu khi qut v chuyn su v ni dung ny c cng b, chng hn [Garry05, Grube09, HGEK07, Yao03, HZ10, GH06, ZZNS09]. Chng 2 s gii thiu chi tit hn v o hp dn.

Nhng iu trnh by trn y cho thy vai tr ca h thng KDD cng nh vai tr ca ngi s dng trong mt phin lm vic ca mnh, to nn s cng tc gia ngi s dng v h thng KDD. Trong s cng tc , h thng KDD to thun tin cho ngi s dng c cch thc linh hot dng cc ngng c cung cp "tri thc" t h thng ph hp vi nhng d on ch quan ca mnh. Nh vy, c th thy rng, cng dng mt phn mm KDD song mi ngi s dng li c th khai thc n theo cch thc ring ca mnh.

Khi phn tch ni dung ba cun sch hng u v khai ph d liu vo nm 2003, Z.H Zhou [Zhou03] cho bit s khc bit khng nh v ni dung khi nim pht hin tri thc t d liu ca ba nhm tc gi (J. Han v M. Kamber; IH Witten v E. Frank; D. Hand, H. Mannila v P. Smyth) u l cc chuyn gia hng u v khai ph d liu v pht hin tri thc trong CSDL. Ti liu ny tip nhn quan nim ca Fayyad, Piatetsky-Shapiro, Smyth (c Z.H Zhou gi l quan nim truyn thng) coi KDD l mt qu trnh bao gm nhiu bc thc hin (xem Khi nim 1.1,), trong , khai ph d liu l mt bc thc hin chnh yu. Cch hiu nh vy quy nh c s phn bit gia hai khi nim khai ph d liu v KDD.

Khi nim 1.2. (Frawley, Piatetski-Shapiro v Matheus [FPS96])

Khai ph d liu l mt bc trong qu trnh Pht hin tri thc trong c s d liu, thi hnh mt thut ton khai ph d liu tm ra cc mu t d liu theo khun dng thch hp.

37 38

Cng v khi nim khai ph d liu, theo B.Kovalerchuk v E.Vityaev [KV01], Friedman tng hp mt s quan nim lin quan sau y:

- Qu trnh khng tm thng nhn bit t d liu ra cc mu c gi tr, mi, hu dng v hiu c (Fayyad),

- Qu trnh trch lc cc thng tin cha bit trc, c th nhn thc c, c th tc ng c t CSDL ln v s dng chng to ra quyt nh cng tc (Zekulin),

- Tp cc phng php c dng trong qu trnh pht hin tri thc nhm tng minh cc quan h v cc mu cha bit trc cha trong d liu (Ferruzza),

- Qu trnh h tr quyt nh khi tm kim nhng mu thng tin cha bit v hu ch t CSDL ln (Parsaye).

Z.H Zhou [Zhou03] gii thiu ba tip cn sau y v ni dung khi nim khai ph d liu qua phn tch ni dung ba cun sch nu trn:

- Qu trnh khm ph tri thc th v t lng ln d liu c lu tr trong CSDL, hoc kho d liu, hoc cc kho thng tin khc (J. Han v M. Kamber),

- S khai thc thng tin tim n, trc cha bit, v c kh nng hu ch t d liu (IH Witten v E. Frank),

- Phn tch tp d liu quan st (thng ln) tm ra cc mi quan h tng minh v tm tt d liu theo cch mi chng va d hiu va hu ch cho ch s d liu (D. Hand, H. Mannila, P. Smyth).

1.2.2. Qu trnh pht hin tri thc trong c s d liu

Qu trnh pht hin tri thc trong c s d liu c m t trong Hnh 1.7 v trnh by chi tit hn trong Hnh 1.8. Tng ng vi s m t chi tit qu trnh KDD (Hnh 1.8), cc nhm bc thc hin

sau y c tin hnh trong qu trnh pht hin tri thc trong CSDL:

Hnh 1.8. Mt m t chi tit qu trnh KDD

(1) M rng hiu bit v min ng dng, v cc tri thc vi u tin thch hp v v mc ch ca ngi dng cui. C th coi ni dung cng vic ny tng ng vi ni dung kho st bi ton trong qu trnh xy dng mt h thng thng tin ni chung.

Mt nhim v quan trng ca bc ny l xc nh bi ton khai ph d liu. Mc 1.6 s gii thiu hai lp bi ton khai ph d liu in hnh nht l m t v d bo v cc bi ton khai ph d liu in hnh thuc vo hai lp ny.

Khi to tp d liu ch, to kho d liu: chn tp d liu v/hoc hng trng tm ti tp con cc bin hoc mu d liu m trn cng vic pht hin tri thc c tin hnh. Tri thc min ng dng c c thng qua vic m rng hiu bit v min ng dng

39 40

ni trn ng vai tr l nn tng tri thc khi to tp d liu ch, kho d liu.

Chng 2 s tho lun chi tit v vai tr ca tri thc v bi ton pht hin tri thc trong mt min ng dng.

(2) Tin x l d liu: thc hin cc thao tc c s nh gii quyt thiu vng gi tr, loi b nhiu hoc yu t ngoi lai, kt ni cc thng tin cn thit ti m hnh hoc loi b nhiu, quyt nh chin lc nhm nm bt cc trng d liu (cc thuc tnh), tnh ton dy thng tin thi gian v s bin i c nh trc.

Cht lng ca h thng khai ph d liu ph thuc vo cht lng ca d liu u vo. Mc tiu ca lm sch d liu nhm m bo d liu u vo c cht lng tt.

Thu gn v trnh din d liu c mc tiu tm c cc c trng hu ch nhm trnh by mi ph thuc d liu theo mc ch ca bi ton. Thu gn d liu c thi hnh v chiu ngang (gim s lng i tng), chiu dc (gim s lng trng d liu) hoc c hai nhm lm cho kch thc d liu c x l, tng tc hot ng ca h thng. S dng cc phng php thu gn hoc bin i chiu nhm rt gn s lng cc bin cn quan tm hoc tm ra cc m t bt bin i vi d liu nhm trnh din d liu ph hp nht. Do khi lng d liu trong bi ton KDD l rt ln cho nn vic thi hnh bc ny l rt cn thit. Khi thu gn theo chiu ngang cn lu l tp d liu c chn la sau khi thu gn phi c tnh i din cho tp ton b d liu ca min ng dng. Vic chn la d liu vo xy dng m hnh khai ph d liu (xy dng nh kho d liu) thng thng cn c tin hnh theo mt phng php m bo tnh "ngu nhin" khi chn la d liu trong min ng dng. Tng t, khi thu gn theo chiu dc cn lu cc thuc tnh cn li m bo tnh i din cho i tng trong bi ton khai ph d liu ang xem xt. Trong khng t bi ton khai ph d liu, khi thu gn theo chiu dc li nhn c kt qu tt hn khng ch v thi gian v khng

gian m cn c v cht lng ca bi ton khai ph d liu khi t c chnh xc cao hn v loi b c mt s thuc tnh gy nhiu. Phng php phn t chnh (Principal Component Analysis: PCA, xem chng 3) thng c s dng trong bi ton thu gn theo chiu dc.

Chng 3 s tho lun cc ni dung chi tit v cc bi ton tin x l d liu v mt s phng php in hnh gii quyt cc bi ton ny.

(3) Khai ph d liu bao gm ba ni dung l la chn bi ton v phng php khai ph d liu thch hp v thi hnh thut ton khai ph d liu.

La chn bi ton khai ph d liu quyt nh mc tiu ca qu trnh KDD l loi bi ton khai ph d liu c th no, chng hn nh bi ton phn lp, hi quy, phn on... Tri thc min ng dng thu nhn thm c t bc m rng hiu bit v min ng dng rt cn thit cho vic la chn bi ton khai ph d liu.

Chn la thut ton khai ph d liu: la chn phng php v thut ton c dng tm mu trong d liu. Ni dung ny bao gm c vic quyt nh cc m hnh v tham s c th c chp nhn v thut ton khai ph d liu ph hp vi tiu chun tng th ca qu trnh KDD.

Thi hnh thut ton khai ph d liu: tin hnh vic d tm cc mu cn quan tm di dng trnh by ring bit hoc mt tp cc trnh by nh quy tc phn lp, cy, hi quy, phn on... Trong bc ny, s h tr ca ngi dng vn ng mt vai tr quan trng. Cc chng 4-7 ca gio trnh ny trnh by cc phng php cho cc khai ph d liu c bn nht.

Mc 1.2.3 trnh by mt s ni dung chi tit hn v bc khai ph d liu trong qu trnh pht hin tri thc t d liu.

41 42

(4) Gii thch mu i vi cc mu c khm ph, c th quay v mt cch hp l ti bt k bc no t bc u tin ti bc thi hnh thut ton khai ph d liu thc hin lp.

(5) Hp nht cc tri thc c khm ph, kt hp cc tri thc ny thnh mt h thng trnh din hoc c bin son d dng v kt xut thnh nhng thnh phn hp dn. Kim tra v gii quyt xung t i vi tri thc c trch chn.

Trong qu trnh pht hin tri thc trong cc CSDL nh c m t trn, chng ta nhn thy c s tham gia ca cc kho d liu (Data Warehouse).

M hnh khai ph d liu ngy cng c tin ha theo nh hng h tr chin lc pht trin ca t chc, ni ring trong cc doanh nghip, m hnh khai ph d liu c tch hp vi m hnh kinh doanh. Chng 2 s trnh by qu trnh tin ha ca m hnh khai ph d liu.

1.2.3. Bc khai ph d liu trong qu trnh pht hin tri thc t d liu

Trong qu trnh pht hin tri thc t d liu, khai ph d liu l bc thc hin chnh yu c nhim v to ra cc mu mi t d liu c tin x l v chuyn dng.

Vic chn la bi ton khai ph d liu no c xc nh ch yu t bc m rng hiu bit v min ng dng. Kt qu tin x l d liu cung cp thm thng tin lm r v bi ton khai ph d liu c xc nh.

i vi bi ton khai ph d liu c chn, tn ti nhiu thut ton gii quyt. V c bn, hiu bit min ng dng v tin x l d liu c bn nh hnh c thut ton (hoc s kt hp ca mt nhm thut ton) c tin hnh, trong vic chuyn dng d liu l hng ti thut ton hay nhm thut ton ny. Cc chng 5-

8 s trnh by mt s thut ton in hnh cho tng loi bi ton khai ph d liu. Chng 10 gii thiu mt vi ch dn lin quan ti cch s dng cc thut ton trong bi ton khai ph d liu.

1.2.4. Kin trc mt h thng khai ph d liu

Kin trc in hnh ca mt h thng khai ph d liu c trnh by trong Hnh 1.9 [HK0106]. Trong kin trc h thng ny, cc ngun d liu cho cc h thng khai ph d liu bao gm hoc C s d liu, hoc Kho d liu, hoc World Wide Web, hoc kho cha d liu kiu bt k khc, hoc t hp cc kiu lit k ni trn.

Hnh 1.9. Kin trc in hnh h thng khai ph d liu [HK0106]

43 44

C s tri thc, bao cha cc tri thc min ng dng hin c, c s dng trong thnh phn h thng khai ph d liu lm tng tnh hiu qu ca thnh phn ny. Mt s tham s ca thut ton khai ph d liu tng ng s c tinh chnh theo tri thc min sn c t c s tri thc trong h thng. C s tri thc cn c s dng trong vic nh gi cc mu khai ph c xem chng c thc s hp dn hay khng, trong c vic i chng mu mi vi cc tri thc c trong c s tri thc. Nu mu khai ph c l thc s hp dn th chng c b sung vo c s tri thc phc v cho hot ng tip theo ca h thng. Nh vy, ngun tri thc b sung vo c s tri thc y khng ch t lp lun lgic theo cc h ton lgic c tri thc mi, khng ch do con ngi hiu bit thm v th gii khch quan b sung vo m cn l tri thc c pht hin mt cch t ng t ngun d liu.

1.3. Khai ph d liu v x l CSDL truyn thng

Nh gii thiu, khai ph d liu l mt th h pht trin mi trong thi gian gn y ca cng ngh CSDL. iu c ngha l c mi quan h gn gi gia bi ton khai ph d liu v bi ton x l (tc nghip) CSDL truyn thng trong mi lin quan ti mt i tng chung l CSDL. Tuy nhin, hai bi ton ny cng c s phn bit. Du hiu phn bit u tin gia khai ph d liu v x l CSDL truyn thng l i tng tc ng ca bi ton khai ph d liu phi l cc CSDL, cc kho d liu c dung lng rt ln, trong khi bi ton tc nghip CSDL truyn thng lin quan ti cc CSDL vi mi kch thc. Thm na, nhng ni dung di y cung cp thm cc thng tin b sung v bi ton khai ph d liu [KV01]. Mi quan h gia h thng qun tr CSDL vi h thng khai ph d liu c m t trong Hnh 1.10 [KV01].

Hnh 1.10. Mi quan h gia h thng CSDL v h thng khai

ph d liu [KV01]

H qun tr CSDL truyn thng c nh hng vic tm kim ti:

- Ghi nhn ring l, chng hn nh cn tm kim cu tr li cho truy vn "Hy hin th s tin ca ng Nguyn Vn A c trong ngy 5 thng Ging nm nay". Vic tm kim cc ghi nhn ring l thng c ch dn l x l giao dch trc tuyn (on-line transaction processing - OLTP).

- Ghi nhn thng k, chng hn nh tr li cu hi "C bao nhiu nh u t nc ngoi mua c phiu X trong thng trc ?". Vic tm kim ghi nhn thng k thng c ch dn l h thng h tr quyt nh thng k (stastical decision suppport system - DSS).

45 46

- Ghi nhn v d liu a chiu, chng hn nh p ng yu cu "Hin th mi c phiu trong CSDL vi mnh gi tng". Vic tm kim cc ghi nhn d liu a chiu thng c hiu l cung cp x l phn tch trc tuyn (on-line analytic processing - OLAP) v x l phn tch trc tuyn quan h (relational OLAP - ROLAP).

cc loi truy vn (nh nhng truy vn ni trn) t ra c vn cn gii quyt mt cch ng n, v qua to ra c cc quyt nh hu ch th cn phi cng nhn tn ti mt gi thit v tri thc min phc hp "y " (sophisticated domain knowledge) m cc loi truy vn ni trn c a ra da trn c s tri thc min . Trong CSDL quan h th tp rng buc, in hnh l tp ph thuc hm cng cc lut suy din Armstrong l mt b phn ca tri thc min ng dng ni trn. Tuy nhin, vi cc CSDL ln c dung lng ti hng trm Gigabytes (GB) th rt kh khn cng nhn mt tri thc min phc hp y .

V mc tiu ca h thng, phng php khai ph d liu h tr vic m rng mc tiu ca CSDL truyn thng bng cch cho php tm kim cc cu tr li cho cc truy vn tuy th s song li quan trng, c tc dng ci tin min tri thc (trong trng hp ny tri thc min phc hp c coi l cha y ) nh:

- Cc c phiu tng gi c c trng g ?

- T gi US$ - DMark c c trng g ?

- Hy vng g v c phiu X trong tun tip theo ?

- Trong thng tip theo, s c bao nhiu on vin cng on khng tr c n ca h ?

- Nhng ngi mua sn phm Y c c trng g ?

- Ti nn mua loi t no ?

- Ti nn vo trng i hc no ?

- Nhng bi bo nn tng v ch nghin cu sinh ca ti l nhng bi bo no ?

- v.v.

Tr li cc truy vn ny dng nh l chng ta khm ph ra c cc quy tc (lut) tim n trong d liu v trn c s cc quy tc m a ra c cc d bo. Nh vy mc tiu ca khai ph d liu l cung cp thng tin, tri thc h tr quyt nh thng qua cc mu, cc lut c khm ph. Cc mu (lut) c khm ph l khng tuyt i, khng mang tnh "bt di bt dch" m c tnh cht "a s trng hp l ng" v c th thay i t thi im ny n thi im khc. Chng hn nh lut kt hp "c n 80% ngi nu mua bia th cng mua thm t tr em" c pht hin cho thy ti thi im ang xem xt phn ng ngi mua bia th cng mua thm t tr em. Pht hin ny c gii thch nh sau. Ti mt s vng phng Ty, ngi chng thng c phn cng trng con nh trong khi ngi v lm vic gia nh. ngi chng thun tin nht khi trng con tr th b c ng bm tr em cn ngi chng ngi ung bia v chi vi con. C th n thi im no khc trong tng lai ca cc vng dn c ni trn hoc ti cc vng dn c khc, khi m th hiu ca ngi n ng trng tr c s thay i, theo h s khng mua bia na th trong c s d liu giao dch s khng tim n "lut" ni trn na.

Nh vy, trong khai ph d liu th gi thit bit v mt tri thc min phc tp "y "l khng cn l yu t ct li, v qu trnh pht hin tri thc c tc dng b sung thm cc tri thc "mi" vo min tri thc . Tnh cht khng y ca tri thc min cho php tri thc min c th c sn, c th c b sung, thay i nh qu trnh pht hin tri thc t d liu.

47 48

1.4. Mt s lnh vc ng dng khai ph d liu in hnh

Theo J. Han v M. Kamber [HK0106], ng dng ca KDD c chia thnh hai lp chnh bao gm lp cc ng dng phn tch d liu - h tr quyt nh v lp cc lnh vc ng dng khc.

Lp cc ng dng trong phn tch d liu v h tr quyt nh bao gm cc ng dng trong phn tch v qun l th trng, phn tch v qun l ri ro, khm ph ngoi lai v cc mu khng hu ch. D liu trong cc ng dng ny l kh phong ph c c t cc giao dch th tn dng, nghin cu i sng cng ng...

Mt s mc tiu khai ph d liu l nh tm ra cc nhm khch hng nh hng tip th da trn cc c trng v nim hng th, mc thu nhp cng nh phn tch th trng cho nh tm ra cc mi lin kt, ng quan h trong vic bn hng d bo theo cc kt hp .

Mt s ng dng in hnh nht l phn tch hng khch hng theo tng loi sn phm nh hng tip th ph hp, phn tch nhu cu khch hng, nh danh loi sn phm thch hp cho tng lp khc hng a ra chin lc kinh doanh i vi nhm khch hng mi, a ra cc bo co tm tt a chiu cng nh nhng thng tin tm tt v mt thng k...

Ngoi ra, ng dng trong lp k hoch ti chnh v nh gi lu lng tin t trong ti chnh ngn hng cng c pht trin. Trong cng tc lp k hoch ti nguyn cng xut hin nhiu ng dng ca KDD. Hn na, c nhiu cch tip cn khc nhau nhm pht hin tri thc c s dng trong cc ng dng nh vy.

Trong nhm phn tch d liu v h tr quyt nh, KDD cn c ng dng kh rng ri trong lnh vc bo him y t, phc v th tn dng, vin thng, th thao, chinh phc v tr.

Lp cc lnh vc ng dng in hnh khc bao gm khai ph Text, khai ph Web, khai ph d liu dng, khai ph d liu sinh hc Mt s sn phm in hnh v khai ph Text v khai ph Web c khng nh c tnh hiu qu, chng hn cc sn phm TextAnalyst*, TextracterTM, WebAnalyst v PolyAnalyst... ca cng ty Megaputer9, hoc WebFountain ca IBM

S pht trin nhanh chng ca khai ph d liu lm cho min ng dng lnh vc ngy cng thm phong ph v a dng, chng hn quan nim ca J. Han v M. Kamber v cc khu vc ng dng khai ph d liu c s thay i t phin bn 2001 ti phin bn 2006 [HK0106]. Trong phin bn 2006, J. Han v M. Kamber coi rng cc lnh vc in hnh ca khai ph d liu l phn tch d liu ti chnh, cng nghip bn l, cng nghip truyn thng, phn tch d liu sinh hc, ng dng cc ngnh khoa hc khc, s xm nhp sai tri...

Cn theo Gregory Piatetsky-Shapiro [Pia06], cc min ng dng in hnh ca khai ph d liu l:

- ng dng trong khoa hc nh thin vn hc, tin sinh hc, y hc (sng ch cc dc phm)...

- ng dng trong thng mi nh qun l quan h khch hng (Customer Relationship Management: CRM), pht hin gian ln, thng mi in t, sn xut, th thao/gii tr, dch v vin thng, tip th nh hng, bo him y t...

- ng dng trong World Wide Web nh my tm kim, qung co trc tuyn, khai ph web v khai ph text...

- ng dng trong hot ng chnh quyn nh pht hin ti phm, pht hin la o thu thu nhp c nhn...

9 http://www.megaputer.com/

49 50

Bng 1.2. Cc ng dng khai ph d liu ni bt (S trong ngoc l s

ngi b phiu tng nm: mt ngi c th lm nhiu ngdng)

Khai ph d liu l lnh vc ng dng c s pht trin nhanh, thch hp vi s pht trin xu th "bng n d liu" v xu th bin ng ca nhu cu x hi. Chng hn, trong thi gian gn y, mng x hi (social network) v phng tin x hi (social media) tr thnh mt trong nhng lnh vc ng dng ni bt ca khai ph d liu. Bng 1.2 cho bit tnh hnh v cc ng dng khai ph d liu ni bt qua thm d ti trang web KDnuggets10. Khuynh hng pht trin ng dng ca khai ph d liu s c trnh by ti Chng 10.

1.5. Kiu d liu trong khai ph d liu

Bng 1.3 cho bit tnh hnh v cc kiu d liu c khai ph qua thm d ti trang web KDnuggets 11. V nguyn l chung, ngun d liu c s dng tin hnh khai ph d liu nhm pht hin tri thc l rt phong ph v a dng, trong in hnh nht l CSDL quan h, kho d liu, CSDL giao dch, cc h thng d liu v thng tin m rng khc.

1.5.1. C s d liu quan h

Th nht, tnh ph bin ca h thng CSDL quan h hin nay to ra mt h qu t nhin quy nh CSDL quan h l mt ngun u vo in hnh nht, c quan tm trc ht ca khai ph d liu. Th hai, mt trong nhng mu c quan tm l mu v cc loi "quan h" m vi bn cht ca mnh, h thng CSDL quan h tim n cc mu dng nh th. Nh bit trong l thuyt CSDL, h thng CSDL quan h thng bao gm mt tp cc bng (hai chiu dc v ngang). Theo chiu dc, bng gm mt s ct (cn c gi l thuc

10 http://www.kdnuggets.com/polls/ 11 http://www.kdnuggets.com/polls/

51 52

tnh, trng hay c trng) v theo chiu ngang bng cha mt tp rt ln cc dng (cn c gi bn ghi hay b). S lng ct ca bng cn c gi l s chiu. H thng CSDL quan h cn bao gm mt m hnh ng ngha m thng thng l m hnh thc th - quan h.

Bng 1.3. Kiu d liu c khai ph (S trong ngoc l s ngi b

phiu tng nm: mt ngi c th dng nhiu kiu d liu)

1.5.2. Kho d liu

Theo J. Han v M. Kamber, tn ti nhiu cch hiu v kho d liu, nhng cch hiu ph dng nht l theo nh ngha ca W.H. Inmon, mt chuyn gia hng u v kho d liu. Theo W.H. Inmon [Inm02], "kho d liu l tp hp cc d liu nh hng theo ch , c tch hp li, c tnh phin bn theo thi gian v kin nh c dng h tr vic to quyt nh qun l". Tn gi ca bn thuc tnh "nh hng theo ch ", "c tch hp li", "c tnh phin bn theo thi gian" v "kin nh" trn y ca kho d liu mi ch cung cp mt s nt c bn nht v cc c trng ca kho d liu. W.H.

Inmon (cng nh J. Han v M. Kamber) gii thch ni dung chi tit v bn thuc tnh ny.

Kho d liu l mt kt qu xut hin trong qu trnh tin ha cc h h tr quyt nh. Thut ng "to kho d liu" (Data warehousing) c dng ch qu trnh xy dng v s dng kho d liu. Nh vy, qu trnh pht hin tri thc trong CSDL tip nhn u vo l cc h thng CSDL, cc nh kho t chc d liu t cc ngun v cc d liu m t. Cn ch rng, p ng bn thuc tnh trn y kho d liu c coi ch bao gm cc d liu c coi l "c cht lng" thng qua cc khu chn la, tin x l v c th bao gm c khu chuyn dng trong qu trnh pht hin tri thc trong CSDL (Hnh 1.4).

Cc nghin cu v trin khai lin quan ti kho d liu ch dn khuynh hng hin ti ca cc h thng thng tin qun l (MIS: Managment Information Systems) ph bin l nhm vo vic thu thp, lm sch d liu giao dch v to cho chng linh hot khi tm kim trc tuyn. Mt tim cn ph bin i vi phn tch kho d liu gi l OLAP (On-Line Analytical Processing), thng qua mt tp cc nguyn l c Codd xut vo nm 1993. Cc b cng c OLAP ch trng ti vic cung cp ti SQL cc tin ch phn tch d liu a chiu cht lng cao bng cc tnh ton gin lc v phn tch nhiu chiu. C pht hin tri thc ln OLAP c coi l hai kha cnh quan h mt thit nhau c tch hp trong mt th h mi cc b cng c trch lc v qun l thng tin.

ng thi vi s pht trin ca cng ngh kho d liu, cc h thng tch hp cc ngun d liu c d liu trong qu kh ln d liu tc nghip c xy dng. Nhiu h thng khai ph d liu c u vo t siu d liu (metadata) cng cc d liu ngun trong cc kho d liu.

53 54

1.5.3. C s d liu giao dch

Mt lp bi ton khai ph d liu ph bin l khai ph quan h kt hp, trong in hnh l bi ton khai ph lut kt hp, c xut pht t vic xem xt cc CSDL giao dch (bn hng). D liu giao dch chnh l d liu nguyn thy xut hin trong nh ngha v lut kt hp cng vi cc o ca lut nh h tr v tin cy. Khi m rng d liu t d liu giao dch sang d liu v hng hoc d liu phc tp hn c trong cc CSDL quan h, cc gii php khai ph lut kt hp c ci tin thch ng vi s bin i ny (bao gi bc chuyn dng d liu trong qu trnh pht hin tri thc t cc CSDL).

1.5.4. Cc h thng d liu m rng

Trong qu trnh pht trin, cc phng php v thut ton khai ph d liu thch hp i vi cc CSDL m rng v cc kiu kho cha d liu c xut. Cc phng php v thut ton ny c ph hp vi d liu trong CSDL hng i tng, CSDL khng gian-thi gian, CSDL tm thi, d liu chui thi gian (bao gm d liu ti chnh), d liu dng, CSDL Text v CSDL a phng tin, CSDL hn tp v CSDL tha k, v World Wide Web.

H thng CSDL quan h - i tng c th c coi l s b sung theo tip cn hng i tng ti cc h thng CSDL quan h. M hnh d liu quan h - i tng m t ng ngha ca h thng CSDL quan h - i tng, c pht trin t m hnh quan h vi vic b sung cc kiu d liu giu ng ngha. Thc th t m hnh quan h thc th c pht trin thnh i tng trong m hnh quan h i tng. khai ph d liu i vi CSDL quan h - i tng.

1.6. Cc bi ton khai ph d liu in hnh

Khai ph d liu l lnh vc nghin cu mang tnh thc tin cao, ng thi li i hi mt nn tng ton hc mnh trong vic xy dng cc m hnh ton hc ph hp nht cho min d liu ca bi ton ang c quan tm. Bc khai ph d liu trong qu trnh KDD thng p dng mt phng php khai ph d liu c th, lin quan n cc khi nim mu v m hnh. Nh c gii thiu trong mc 1.1, mu l mt biu thc trong mt ngn ng m t L no c chn. M hnh c coi l mt biu thc tng qut trong ngn ng m t L ni trn; tnh tng qut ca m hnh c th hin thng qua cc tham s m hnh, trong trng hp , mt mu l mt th hin ca m hnh. Chng hn, biu thc ax2 + bx (vi hai tham s a v b) l m hnh cn 3x2 + x l mt mu trong m hnh (i vi mu ny th cc tham s m hnh a v b c cho gi tr c th, a=3 v b= 1).

Nhim v ca bi ton khai ph d liu t mt tp d liu quan st (tp cc s kin) c th hoc cn phi xc nh m hnh ph hp vi tp d liu quan st , hoc cn tm ra cc mu t tp d liu .

Bi ton khai ph d liu thng hng ti mt trong hai loi m hnh l m hnh theo tip cn thng k (m hnh thng k) hoc m hnh lgic. M hnh thng k c nh hng ti loi m hnh bao hm cc yu t cha xc nh, chng hn nh m hnh ax + e, trong m hnh ny th x l bin trong ngn ng m t L, cn e c th l bin ngu nhin Gauss (th hin tnh cha xc nh ca m hnh). Ngc li, m hnh lgic nh hng ti loi m hnh xc nh hon ton, chng hn ax, trong khng tha nhn yu t khng r rng khi m hnh ha. M hnh thng k c dng hu khp i vi cc ng dng khai ph d liu thc t.

55 56

Hu ht cc phng php khai ph d liu c xy dng c ni dung t cc phng php hc my, thit k mu v thng k (phn lp, phn on, m hnh th). Thut ton gii quyt mi bi ton ni trn cun ht mt phm vi ngi quan tm a dng bao gm c cc chuyn gia phn tch d liu ln nhng ngi cha h c kinh nghim.

mc cao - tng qut, hai mc tiu ch yu ca khai ph d liu l d bo v m t, m chng ta coi hai mc tiu ny tng ng vi hai bi ton tng qut ca khai ph d liu. Bi ton d bo s dng mt s bin (hoc trng) trong CSDL d on v hoc gi tr cha bit ( c) hoc gi tr s c trong tng lai ca cc bin. Bi ton m t hng ti vic tm ra cc mu m t d liu. D on v m t c tm quan trng khc nhau i vi cc thut ton khai ph d liu ring. Trong ng cnh KDD th vn m t c khuynh hng quan trng hn vn d bo, v iu ny l tri ngc vi ni dung ch yu ca cc ng dng nhn dng mu v hc my th vn d bo l quan trng hn. iu c v tri ngc c th c gii thch khi xem xt, phn tch ni dung ca chnh khi nim "pht hin tri thc trong CSDL"; khi nim ny bao hm tnh hung sn c d liu pht hin cc mu tim n trong d liu , cc mu tim n lin quan ti bi ton m t d liu. Mt khc, m t c m hnh d liu th cng rt thun tin cho d bo.

mc chi tit - c th, d bo v m t c th hin thng qua cc bi ton c th nh m t khi nim, quan h kt hp, phn cm, phn lp, hi quy, m hnh ph thuc, pht hin bin i v lch, v mt s bi ton c th khc nh trnh by di y.

1.6.1. M t khi nim

Ni dung ca bi ton m t khi nim (concept description) l tm ra cc c trng v tnh cht ca khi nim "m t" khi nim

. in Hnh nht trong lp bi ton ny l cc bi ton nh tng qut ha, tm tt, pht hin cc c trng d liu rng buc.

Bi ton tm tt l mt bi ton m t in hnh, p dng cc phng php tm ra mt m t c ng i vi mt tp con d liu. Mt v d in hnh v bi ton tm tt l bi ton tnh k vng v lch chun ca mt tp d liu trong thng k xc sut; hai gi tr ny chnh l hai c trng in hnh nht v mt hin tng c dy gi tr th hin m chng ta quan st c.

Nhiu phng php c bin lun i hi vic thu nhn c cc quy tc tm tt, k thut hin th a bin, pht hin quan h hm gia cc bin. K thut tm tt thng c p dng trong phn tch d liu tham d c tng quan v t ng ha sinh ra cc thng bo.

Trong khai ph Text v khai ph Web, tm tt vn bn l mt biu hin c th ca tm tt, theo t mt vn bn c, cn tm ra vn bn ngn gn (vi di 100 t, 200 t hoc 500 t) m vn gi c ng ngha c bn ca vn bn gc.

1.6.2. Quan h kt hp

Pht hin mi quan h kt hp (associative relation) trong tp d liu l mt bi ton quan trng trong khai ph d liu. Mt trong nhng mi quan h kt hp in hnh l quan h kt hp gia cc bin d liu, trong bi ton khai ph lut kt hp (associative rule) l mt bi ton in hnh. Bi ton khai ph lut kt hp (thuc lp pht hin quan h kt hp), thc hin vic pht hin ra mi quan h gia cc tp thuc tnh (cc tp bin) c dng XY, trong X, Y l hai tp thuc tnh. V Hnh thc, lut kt hp c dng ging nh ph thuc hm trong CSDL quan h, tuy nhin, n khng c nh sn t tri thc min.

Trong khai ph text v khai ph web tn ti nhiu bi ton pht hin quan h kt hp, in hnh nh bi ton pht hin quan h ng

57 58

ngha (chng hn nh quan h nhn-qu, quan h ton b - b phn, quan h chung-ring...) trong vn bn (hoc trong tp vn bn), bi ton pht hin mi quan h gia ni dung trang web ngi s dng ang quan tm ti cc trang web m h c th s hng ti...

1.6.3. Phn lp

Phn lp (Classification/Categorization) thc hin vic xy dng (m t) cc m hnh (hm) d bo nhm m t hoc pht hin cc lp hoc khi nim cho cc d bo tip theo. Mt s phng php in hnh l cy quyt nh, lut phn lp, mng neuron. Ni dung ca phn lp chnh l hc mt hm nh x cc d liu vo mt trong mt s lp bit. V d, phn lp mt vn bn (bao gm c trang web) vo mt trong mt s lp vn bn (trang web) bit, phn lp khuynh hng trong th trng ti chnh, pht hin t ng cc i tng ng quan tm trong CSDL nh ln.

Hnh 1.11. S biu din m hnh hc my: cn hc nh x biu din bng ng lin nt xin [KV01] (Lu , hc khng gim st khng c gi tr mc tiu cho v d hc: khng c ng lin nt)

Hnh 1.11 m t s b v bi ton phn lp (thng c tng ng vi hc c gim st), theo ng ngang lin nt cho bit bit thuc tnh lp i vi mt tp hp d liu no (tp d liu hc). Ni dung chi tit hn v bi ton phn lp s c trnh by chi tit hn trong cc chng sau.

1.6.4. Phn cm

Phn cm (Clustering) thc hin vic nhm d liu thnh cc "cm" (c th ci l cc lp mi) c th pht hin c cc mu phn b d liu trong min ng dng. Phn cm l mt bi ton m t hng ti vic nhn bit mt tp hu hn cc cm hoc cc lp m t d liu. Cc cm (lp) c th tch ri nhau v ton phn (to nn mt phn hoch cho tp d liu) hoc c trnh by p hn nh phn lp c th bc hoc c th chng ln nhau (giao nhau). V d nh bi ton pht hin cc nhm ngi tiu dng trong CSDL tip th hoc nhn bit cc loi quang ph trong tp php o khng gian hng ngoi... Thng thng, mc tiu nh hng ca bi ton phn cm l cc i tnh tng ng gia cc phn t trong mi cm v cc tiu tnh tng ng gia cc phn t thuc cc cm khc nhau.

Trong nhiu trng hp, phn cm cn c gi l hc my khng gim st (unsupervised learning) v phn lp cn c gi l hc my gim st (supervised learning). S b v m hnh hc my (c gim st v khng gim st) c din t nh ti Hnh 1.11 [KV01]. Tuy cng s dng hc my nh phn lp thuc loi khai ph d liu d bo cn phn cm thuc loi khai ph d liu m t.

Trong mt s ng dng, bi ton phn on (segmentation) cn c gii quyt. V ni dung, phn on l t hp ca phn cm v phn lp, trong phn cm c tin hnh trc v sau l phn lp.

59 60

1.6.5. Hi quy

Hi quy (regresion) l mt bi ton in hnh trong phn tch thng k v d bo, trong tin hnh vic d on cc gi tr ca mt hoc mt s bin ph thuc vo gi tr ca mt tp hp cc bin c lp. M hnh hi quy l kh thng dng trong d bo di hn. Trong khai ph d liu, bi ton hi quy c quy v vic hc mt hm nh x d liu nhm xc nh gi tr thc ca mt bin theo mt s bin khc. Tnh hung ng dng hi quy rt a dng, chng hn nh d on s lng sinh vt pht quang trong khu rng nh o vi sng cc sensor t xa, hoc c lng xc sut ngi bnh c th cht theo kt qu test triu chng, hoc d bo nhu cu ngi tiu dng i vi mt sn phm mi c coi nh mt hm ca qung co tiu dng, hoc d bo chui thi gian m cc bin u vo c coi nh bn tr thi gian ca bin d bo

1.6.6. M hnh ph thuc

Bi ton xy dng m hnh ph thuc hng ti vic tm ra mt m hnh m t s ph thuc c ngha gia cc bin. M hnh ph thuc gm hai mc: mc cu trc ca m hnh m t (thng di dng th) trong cc bin l ph thuc b phn vo cc bin khc, trong khi mc nh lng ca m hnh m t sc mnh ca tnh ph thuc khi s dng vic o tnh theo gi tr s. V d, li ph thuc xc sut cn m bo tnh c lp iu kin nhm nh r din mo cu trc ca m hnh v xc sut hoc tng quan m t sc mnh ca tnh ph thuc. Phn tch khuynh hng v tin ha cng c coi thuc vo loi khai ph m hnh ph thuc. Trong phn tch khuynh hng v tin ha, cc phng php phn tch xu th, khai ph mu k tip, phn tch da trn tnh tng t thng c p dng.

1.6.7. Pht hin bin i v lch

Tp trung vo vic pht hin hu ht s thay i c ngha di dng o bit trc hoc gi tr chun, cung cp nhng tri thc v s bin i v lch cho ngi dng. Bi ton pht hin bin i v lch cn c ng dng trong bc tin x l trong qu trnh pht hin tri thc trong CSDL. Chnh v l do , cn trnh suy ngh cho rng s bin i v lch mang ngha "khng chnh quy" m phi quan nim s bin i v lch (c th l bt thng) l mt ni dung bn cht ca d liu.

Ngoi ra c th k ti bi ton phn tch nh hng mu v mt s bi ton khai ph d liu kiu thng k khc.

1.7. Tnh lin ngnh ca khai ph d liu

KDD nhn c s quan tm c bit ca cc nh nghin cu trong cc lnh vc hc my, thu nhn mu, CSDL, thng k, tr tu nhn to, thu nhn tri thc i vi h chuyn gia c trnh by trong Hnh 1.12 [HK0106]. H thng KDD li cun cc phng php, thut ton v k thut t cc lnh vc ri rc nhau ny. Mc tiu thng nht l trch lc tri thc t d liu trong ng cnh cc CSDL ln. ZH Zhou [Zhou03] nhn nh rng khai ph d liu nhn c s ng gp ca rt nhiu ngnh nh CSDL, hc my, thng k, thu hi thng tin, trc quan ha d liu, tnh ton song song v phn tn. Ba ngnh ng gp chnh l CSDL, hc my, thng k. Trong khai ph d liu, CSDL ng gp cc k thut qun l d liu, hc my ng gp cc k thut phn tch d liu thc tin, v thng k ng gp cc nn tng l thuyt vng chc. Tc gi n d rng khai ph d liu nu khng c s ng gp ca CSDL v hc my s nh tm kim

61 62

trong ng c, nu khng c s ng gp ca thng k s nh xy dng lu i trong khng kh.

Mt s lp lun c trnh by ti cc mc trc (1.2, 1.3) ch dn rng khai ph d liu l bc pht trin mi ca cng ngh CSDL, v vy nhiu ni dung trong khai ph d liu l gn gi vi CSDL [HK0106]. ng thi, mt s du hiu phn bit gia h thng CSDL iu hnh tc nghip truyn thng vi h thng khai ph d liu cng c tho lun; cc du hiu in hnh nht bao gm quan nim v mt gi thit sn c mt tri thc min ng dng y , loi Hnh cc cu hi th hin mc tiu ca h thng v kch thc tp d liu i tng kho st.

Hnh 1.12. Tnh a/lin ngnh ca khai ph d liu

Ti nguyn d liu u vo cho cc h thng khai ph d liu gm c cc CSDL, cc kho d liu v cc loi ngun cha d liu khc. Chnh v l do , trong khng t trng hp, lnh vc kho d liu c coi l mt b phn ca lnh vc khai ph d liu v pht hin tri thc trong CSDL.

i vi cc lnh vc hc my v thu nhn mu, s an xen vi khai ph d liu (v KDD) tri theo cc nghin cu v l thuyt v

thut ton i vi cc h thng trch lc mu v m hnh d liu (ch yu i vi cc phng php khai ph d liu). Cc phng php hc my gim st (phn lp), khng gim st (phn cm), bn gim st (phn lp v phn cm) rt ph bin trong khai ph d liu, nhm la chn m hnh v xc nh tham s m hnh trong cc h thng KDD. Trng tm ca KDD i vi vic m rng cc l thuyt v thut ton hc my hng ti bi ton tm ra cc mu c bit (nhng mu m trong mt s ng cnh cn c gi l tri thc hu dng hoc hp dn) trong cc tp hp d liu c dung lng ln ca th gii thc. Nh vy, khai ph d liu m rng ni dung hc my thng qua cc cng vic la chn d liu u vo, trnh din mu, nh gi mu u ra... trong ng cnh min d liu cn x l c dung lng rt ln.

Cng vi tip cn m hnh lgic, m hnh thng k l tip cn ph bin trong cc bi ton pht hin tri thc trong c s d liu, v vy, chuyn ngnh KDD c rt nhiu im chung vi chuyn ngnh thng k, c bit l phn tch d liu thm d (EDA: Exploratory Data Analysis) cng nh d bo [Fried97, HD03]. H thng KDD thng gn kt vi cc th tc thng k c bit i vi m hnh d liu v nm bt nhiu trong mt khung cnh pht hin tri thc tng th. Cc phng php khai ph d liu da theo thng k nhn c s quan tm c bit to nn lp phng php khai ph d liu rng ln da trn hc my thng k. Robert Nisbet v cng s [NEM09], Trevor Hastie v cng s [HTF09] cung cp cc ni dung kh ton din v b ch v cc phng php hc my thng k v khai ph d liu thng k. Robert Nisbet v cng s trnh by mt cch h thng qu trnh tin ha ca thng k ton hc, bao gm c s pht trin cc ni dung ca thng k ton hc ti khai ph d liu thng k.

V khai ph d liu v x l d liu thng k rt gn gi vi nhau v mt s ni dung trong x l d liu thng k c tch hp vo qu trnh khai ph d liu, tuy nhin, cng cn nu ra mt s

63 64

khc bit gia bi ton thng k ton hc v bi ton khai ph d liu.

u tin, khai ph d liu khc bit vi phn tch thng k trong bi ton thng k ton hc v cc gi nh c bn, trong phn tch thng k yu cu cc iu kin cht ch v phn b d liu, v tham s li trong khi khai ph d liu khng i hi nhng gi nh nh vy. Trong bi ton khai ph d liu, tri thc min tng ng vi gi nh i hi ca phn tch thng k l kt qu ca cng vic tm hiu d liu cng phu m khng phi l sn c theo gi nh. Nh vy, phng php phn tch thng k c th c huy ng trong bc hiu d liu ca qu trnh khai ph d liu.

Th hai, mc tiu ca phn tch thng k l kim th gi thit hoc xc nh tham s, trong khi mc tiu ca khai ph d liu l xc nh m hnh d bo v chnh xc ca m hnh d bo . C th hn, trong bi ton phn tch kim nh gi thit thng k, cho trc mt gi thit thng k th cng vic cn tin hnh l kim tra xem tp hp ton b cc d liu quan st c c ph hp vi gi thit thng k ni trn hay khng, hay cng vy, gi thit thng k c ng trn ton b d liu quan st c hay khng. Nu kim nh cho kt qu khng ph hp c ngha l gi thit thng k l khng ng trn tp d liu quan st. Nh vy, tnh ng n ca gi thit thng k c xem xt trn tp tp d liu quan st c.

Th ba, phn tch thng k coi tp d liu x l l phn ly mu ca tp d liu ton cc trong khi khai ph d liu coi tp d liu cn x l l ton b d liu thuc min ng dng. Trong khai ph d liu, m hnh kt qu khai ph d liu l khng c xc nh trc cn phi ph hp vi tp ton b d liu ca min ng dng m khng phi ch vi tp d liu quan st c (tp d liu quan st c ch l mt b phn m thng l rt nh so vi min d liu ca th gii thc, xem Hnh 1.8) do cn m bo cc tham s m hnh khng ph thuc vo cch chn tp d liu hc. Chnh v l do ct li ny m

bi ton hc khai ph d liu i hi p ng yu cu l tp d liu hc cng nh tp d liu kim tra cn c tnh "i din" cho ton b d liu trong min ng dng v hai tp d liu ny cn c lp nhau. Trong mt s bi ton khai ph d liu, hai tp d liu ny (hoc tp d liu kim tra) c cng b di dng chun.

Th t, phn tch c i hi kh r rng v kch thc tp d liu mu v c tnh cht tnh (n nh), trong khi khai ph d liu tip cn theo hng cng nhiu cng tt, hn na d liu c th ng. Tip theo, khai ph d liu cho php thi hnh lp ci thin m hnh kt qu trong khi vic thi hnh lp c th dn ti kt lun sai lm trong phn tch thng k.

Cui cng, cc thut ng dng trong hai lnh vc nghin cu ny cng l du hiu phn bit chng, chng hn, lnh vc khai ph d liu dng cc thut ng bin ra/bin mc tiu, thut ton khai ph d liu, thuc tnh/c trng, bn ghi... trong khi th lnh vc x l d liu thng k dng cc thut ng tng ng l bin ph thuc, th tc thng k, bin gii thch, quan st...

Nh c trnh by, qu trnh pht hin tri thc lm vic vi tp hp d liu ln m trong nhiu trng hp tp d liu tr nn khng l. Phm vi tc ng to ln v a dng i hi cc thut ton khai ph d liu phi ng n v hiu qu; chnh v iu cho nn rt nhiu thut ton khai ph d liu c xut. ZH Zhou [Zhou03] gii thiu v bn thnh phn ca mt thut ton khai ph d liu l cc m hnh v mu, cc hm nh gi, cc phng php tm kim v ti u ha, v chin lc qun l d liu.

Xindong Wu v cng s [WKQ08] cung cp mt danh sch gm mi thut ton khai ph d liu ni ting nht, l cc thut ton C4.5, k-Means, SVM, Apriori, EM, PageRank, AdaBoost, kNN, Naive Bayes, v CART. Cc tc gi cng gii thiu nhng ni dung c bn nht ca mi trong mi thut ton ni trn. Mt s ni dung c bn

65 66

nht ca hu ht c thut ton trong mi thut ton ny s c gii thiu trong cc chng t 4-7 ca ti liu ny.

Nh c khng nh ti cc phn trc y l khng phi tt cc cc mu u hu dng v h thng cn a ra cc tiu ch lc cc mu c coi l hp dn nht. Thng thng cc h thng s dng mt ngng hp dn cc tiu cho cc mu c coi l tri thc, chng hn trong bi ton pht hin lut kt hp, ngi ta ch gi li cc lut vt qua ngng h tr ti thiu v tin cy ti thiu. Ngay c trong trng hp , khng phi mi tri thc c h thng coi l hu dng u hon ton ph hp vi ngi s dng. Bc trc quan ha trong qu trnh KDD hin th cc tri thc c h thng pht hin mt cch trc quan nht to thun li cho ngi s dng (thng qua tri thc v kinh nghim) la chn ra cc tri thc thc s hu dng cho mc ch ng dng ca ngi s dng.

Pht hin my vi mc tiu l pht hin cc lut kinh nghim t quan st v th nghim v m hnh nhn qu pht hin cc kt lun ca m hnh nhn qu t d liu l nhng lnh vc nghin cu c mi lin h vi nhau.

Khai ph d liu v pht hin tri thc t d liu cng chng kin s thm nhp rng ln ca l thuyt tp m (chng hn, [EM03, HP03, STH06] v cc cng b khoa hc trong dy hi ngh quc t Intenational Conference on Fuzzy Systems and Knowledge Discovery: FSKD12 v mt s hi ngh quc t uy tn khc), l thuyt tp th (chng hn, [Zia94, Ohrn99, SZ00, Li07, NS08, Szczu11] v cc cng b khoa hc ti chui hi ngh quc t "Rough Sets and Knowledge Technology: RSKT13) v l thuyt kt hp tp m th

12 http://icnc-fskd.dhu.edu.cn/ 13 http://rskt.cs.uregina.ca/

[Jenssen11] . Chng 9 trnh by cc ni dung chi tit v khai ph d liu da trn l thuyt tp m, tp th v tp m-th.

Khai ph d liu v pht hin tri thc t d liu l lnh vc nghin cu v ng dng c quan h mt thit vi s pht trin kinh t x hi, v vy, theo thi gian, khai ph d liu a v ang thu ht thm s tham gia ca nhiu ngnh, chuyn ngnh khc khng ch trong lnh vc CNTT m cn cc lnh vc khc.

Cu hi v Bi tp

1.1. Ni dung, ngha nh hng cng nghip v kinh t ca nh lut Moore.

1.2. Phn bit bi ton qun tr C s d liu tc nghip vi bi ton khai ph d liu.

1.3. Phn tch vai tr ca c s tri thc trong mt h thng khai ph d liu.

1.4. Phn bit bi ton khai ph d liu vi bi ton kim nghim gi thit thng k.

1.5. Han v Kamber [HK0106] quan nim khai ph d liu v pht hin tri thc trong CSDL l bc pht trin mi ca cng ngh CSDL. Hy lp lun lm sng t quan nim trn.

1.6. Trnh by mt s mu truy vn trong h thng qun tr c s d liu v h thng khai ph d liu. Phn tch lm sng t cc mu truy vn trong h thng khai ph d liu l phc tp hn mu truy vn trong h thng qun tr CSDL.

1.7. H thng khai ph d liu c nht thit c ngun u vo l kho d liu hay khng ? Phn tch mt s li im khi h thng khai ph d liu c ngun d liu u vo ch l cc kho d liu.

67 68

1.8. Phn tch v tnh "khng tm thng" ca qu trnh pht hin tri thc trong CSDL.

1.9. Phn bit bi ton khai ph d liu m t vi bi ton khai ph d liu d bo.

1.10. Phn tch tm quan trng ca khu lm sch d liu v tin x l d liu trong qu trnh khai ph d liu v trnh by s b v ni dung ca khu ny.

1.11. Phn tch v s cn thit phi tin hnh tnh ton gi tr mt s o no trong cc bi ton khai ph d liu.

Chng 2. Cng ngh tri thc v pht hin tri thc t d liu

Nh c cp ti Chng 1, th gii ngy nay ang chuyn i t kinh t hng ha (good economic) sang kinh t dch v (service economic). Ba khi nim kinh t ni bt l kinh t tri thc, kinh t thng tin v kinh t dch v. Si ch xuyn sut ni dung ba khi nim kinh t ni trn l tri thc. S dng tri thc l ng lc ch cht cho tng trng kinh t quc gia, cng chnh l ng lc ch cht cho tng cng li th cnh tranh ca doanh nghip, t chc. Trong xu th pht trin , CNTT ngy cng khng nh tm quan trng chin lc. c bit, ngnh cng nghip da trn d liu c hnh thnh v ang pht trin vi tc cao. Khai ph d liu v pht hin tri thc trong d liu l nn tng ca ngnh cng nghip da trn d liu.

Chng 1 cng trnh by mt s ni dung khi qut v khai ph d liu v pht hin tri thc t d liu. Chng 2 s gii thiu chi tit v vai tr v ni dung ca cng ngh tri thc m mt ni dung c bn trong l pht hin tri thc t d liu.

2.1. Vai tr ca CNTT trong kinh t tri thc

Nghin cu khoa hc lin lnh vc cho thy pht trin CNTT v pht trin kinh t c mi quan h hu c mt thit, trong cc quc gia c trnh CNTT pht trin cao cng chnh l cc quc gia c nn kinh t pht trin cao. T v th c k vng c phn qu cng iu v m h ban u, CNTT ngy cng khng nh v th chin lc trong pht trin kinh t, trong tng trng hiu qu ca doanh nghip v t chc. Tuy nhin, cn c mt nn tng nhn thc chnh

69 70

xc v ton din v v th chin lc ca CNTT xc nh chin lc pht trin da trn CNTT ng n v ngn nga c cc biu hin sai lch trong nhn thc v vai tr ca CNTT, hoc theo hng ng nhn v lm dng vai tr ca CNTT dn ti lng ph, tham nhng hoc theo hng ph nhn v th chin lc ca CNTT.

u tin, mc con 2.1.1. gii thiu mt s lun im theo hng ph nhn v th chin lc ca CNTT, in hnh l lun im ca Robert M. Solow vo nm 1987 v lun im ca Nicolas Carr vo nhng nm 2003-2004. Tip theo, nhm cung cp mt s ni dung lm sng t v th chin lc ca CNTT, khi nim v kinh t tri thc v vai tr ca CNTT trong kinh t tri thc s c gii thiu trong mc con 2.1.2.

2.1.1. Nghch l hiu qu ca CNTT ca Robert Solow v lun im ca N. Carr

2.1.1.1. Nghch l hiu qu ca CNTT

Vo nm 1987, Robert M. Solow, mt nh kinh t ngi M c tng gii thng Nobel v kinh t, pht biu "Chng ta nhn thy my tnh mi ni ngoi tr trong thng k hiu qu" (nguyn vn: You can see the computer age every where but in the productivity statistics) [Solow87]. Pht biu ny c Erik Brynjolfsson [Bryn93] ch dn nh l "nghch l hiu qu ca CNTT (Productivity Paradox of Information Technology). Theo Erik Brynjolfsson, thng k hiu qu c R. M. Solow lun c trong nghch l hiu qu ca CNTT c din t nh di y.

- Trong bn thp nin (1960- 1990), t l u t cho my tnh ca nc M tnh theo GDP tng nhanh t 0,003 % GDP (thp nin 1960), 0,05% (thp nin 1970s), 0,3% (thp nin 1980s), ti 3,1% (thp nin 1990s) nhng t l tng GDP trung bnh theo nm li gim t 4,5% (thp nin 1960s) xung 2,95% (thp nin 1970s) ri

2,75 (thp nin 1980s) v 2,20% (thp nin 1990s). Tng u t CNTT c v nh khng gp phn vo tng GDP nc M nu khng ni l cn lm gim i.

- Theo thng k t hng trm nghn doanh nghip M, u t CNTT tnh theo u nhn vin v hiu qu kinh doanh cng khng c mi quan h r rng, khng hng ti k vng "u t CNTT tng th hiu qu kinh doanh cng tng". C mt trng hp c bit, ring vi cc doanh nghip trong lnh vc ti chnh ngn hng, hiu qu kinh doanh c quan h t l thun vi u t CNTT.

- Tnh trng u t CNTT mt cch lng ph cng din ra i vi mt b phn c nhn v h gia nh.

E. Brynjolfsson [Bryn93] a ra nhn nh rng thc cht cc hin tng trn y khng thc s l nghch l hiu qu ca CNTT. Tc gi cung cp bn gii thch di y v cc hin tng ni trn:

- Li o lng trong cng thc tnh hiu qu. Li o lng th hin theo hai kha cnh chnh. Cng thc tnh hiu qu ca kinh t c in c mt li ln khi ch o lng ti nguyn trc tip lin quan ti vn, lao ng v gi tr. Trong thi i kinh t tri thc (xu th chuyn i t kinh t hng hng ha sang kinh t hng dch v), cc yu t ti nguyn gin tip (tri thc nhn vin v tri thc doanh nghip, ti nguyn quy trnh t chc ca doanh nghip...) ngy cng ng vai tr quan trng trong pht trin kinh t quc gia v cnh tranh doanh nghip th chng cn phi xut hin trong cng thc tnh hiu qu hin i. Hnh 2.1 biu din mt cng thc o lng hiu qu dch v vi s than gia ca nhiu yu t ti nguyn gin tip c u vo v u ra. Lu rng, o lng ti nguyn gin tip (thuc c u ra ln u vo) li l mt bi ton rt kh.

71 72

Hnh 2.1 Mt cng thc o lng hiu qu.

- Khng ging nh cc khon u t c s h tng, u t CNTT c mt khong thi gian tr pht huy hiu qu. iu ny c nguyn nhn t vic nhn vin trong doanh nghip phi c mt khong thi gian (theo E. Brynjolfsson, thng l 2-3 nm) mi c th s dng thnh tho cc cng c ca CNTT. Tc gi cng khuyn ngh v vic cn thc hin gii php rt ngn tr ny.

- Tnh phn phi li v ti nguyn thng tin. Thng tin v tri thc va l ti nguyn quan trng ca doanh nghip song cng c coi l mt dng "sn phm hng ha cng cng", cho nn, chi ph u t CNTT pht trin ca mt doanh nghip c th b bao gi thm chi ph u t CNTT cho doanh nghip khc. i vi tnh hung ny, cc doanh nghip cn phi m bo c mt yu cu l trong vng i ca thng tin v tri thc ca doanh nghip, chng phi lm li nhiu nht cho chnh bn thn doanh nghip u t.

- Sai lm trong qun l u t CNTT. Cc khon u t CNTT c thi hnh song quyt nh u t chng li c th khng c

nh hng ti li ch ca doanh nghip. Tnh trng ny c nguyn nhn t cc quyt nh u t l lng lo dn ti vic xy dng cc h thng khng hiu qu, hoc n gin l s dng cc chin lc to quyt nh li thi khi quyt nh u t CNTT. S dng chnh cng c CNTT, c bit l cng c khai ph d liu, l mt bin php khc phc c hin tng ny. Lu rng, y khng cp ti mt vn tiu cc x hi l tham nhng trong u t cho CNTT.

Nh vy, ngay t nhng nm u tin ca thp nin 1990, cc nh khoa hc khng nh c rng "nghch l hiu qu ca CNTT" l khng ng trong thc tin. Khng nhng th, vai tr chin lc ca CNTT ngy cng c nhn mnh trong pht trin kinh t tri thc [OECD96]. Tuy nhin, mt vi nh kinh t, in hnh l Nicolas Car, vn bo th v by t mi nghi ng v vai tr chin lc ca CNTT.

2.1.1.2. Lun im ca N. Carr

Vo nm 2003, N. Carr trnh by mt s lun im sau y ph nhn vai tr chin lc ca CNTT [Carr03]:

- CNTT xut hin khp ni v tm quan trng chin lc ca n gim. Cch tip cn u t v qun l CNTT cn phi c thay i mt cch ng k !

- Khi mt ti nguyn ( ni CNTT) tr thnh bn cht cnh tranh nhng khng quan trng cho chin lc th ri ro m n to ra li tr nn quan trng hn so vi cc li th m n cung cp.

- Vi vic nhanh chng bin mt cc c hi t c li th chin lc t CNTT, nhiu doanh nghip cn phi c mt ci nhn nghim khc trong u t vo CNTT v qun l cc h thng ca h.

ng thi, N. Carr a ra ba quy tc hng dn cho tng lai vi nh hng ph nhn vai tr chin lc ca CNTT. Nm 2005, N. Carr li cng b mt bi vit khc [Carr05] nhm cng c cc lun

73 74

im trn y. Lun im ph nhn vai tr chin lc ca CNTT m N. Carr pht biu to ra mt ln sng phn bc mnh m. Chnh v vy, N. Carr lt vo danh sch 100 ngi c tn c nhc n nhiu nht trn th gii.

Sai lm ca N. Carr l ch ng quan nim CNTT nh l mt loi cng ngh h tng (ging nh in nng), t dn n vic khng nhn thc c vai tr chin lc ca CNTT trong pht trin tri thc quc gia cng nh tri thc doanh nghip. Thng qua cc phn tch lin quan ti 11 nhn nh ca N. Carr, Paul A. Strassmann (Executive Advisor, NASA; Former CIO of General Foods, Kraft, Xerox, the Department of Defense, and NASA) lm sng t cc sai lm trong cc bi vit ca N. Carr14.

Tuy nhin, tng t nh gii thch "nghch l hiu qu ca CNTT" t yu km trong qun l u t CNTT, khuyn co v cch thc tip cn u t v qun l CNTT ca N. Carr cng mang ngha tch cc. Di y l mt s nhn nh ca mt s nh qun l lin quan ti khuyn co ny1:

- Nu c mt iu m chng ta hc c t nhng nm 1990 l s khi u da trn CNTT, tng nh mt v n v tr nhng li him khi to ra mt n p tng xng nh k vng. Nh ra phi gip cc doanh nghip hiu rng CNTT ch l mt cng c, cc nh cung cp cng ngh li nhm ti n nh mt thuc bch bnh Mua cng ngh ny i v cc vn ca anh s c gii quyt ! (John Seely Brown, Former Chief Scientist, Xerox Palo Alto, California v John Hagel III, Management Consultant and Author, Burlingame, California).

- Cng vic ca CTO (Chief Of Technical: ngi ng u b phn cng ngh) v CIO (Chief Of Information: ngi ng u v

14 Harvard Bussiness Review, June 2003

thng tin) ca t chc s tr nn quan trng cha tng c trong cc thp nin tip theo. Gi k nng cn thit trong mt t chc s thay i rt nhanh cnh tranh trong thi i thng tin (F. Warren McFarlan, Albert H. Gordon Professor of Business Administration, Harvard Business School, Boston v Richard L. Nolan, William Barclay Harding Professor of Business Administration, Harvard Business School, Boston)

- Ti ng tnh nhiu vi khuyn co ca Nicholas Carr v cch thc cc doanh nghip nn c phn ng vi mt thc t khng th chu ng c l CNTT tr thnh mt loi hng ha. Nhng ti sao Carr li khuyn co cc iu lo lng ti cc nh qun l CNTT ? Phi chng l v cc bi ton lnh o nh qun l v kim sot ri ro v kinh ph t ha hn hoc thch thc hn so vi vic theo ui li th cnh tranh ? CNTT lun lun quan trng l vn trong mi quan nim. CNTT bt buc h tr kinh doanh khng ch bng p dng lgc v cng ngh m cn bng p dng lgic v bn cht chung (Jason Hittleman, IT Director, RKA Petroleum Companies, Romulus, Michigan).

Lin quan ti u t cho CNTT, thng qua vic kho st v u t v hiu qu CNTT ca trn 5700 doanh nghip M, Paul A. Strassmann a ra mt s khuyn ngh [Strass07]:

- C th chi tiu cho CNTT hn hoc km so vi mc trung bnh ca cc doanh nghip ng hng (gi l mc thng thng), nhng v tng th th chi tiu nh th cn a ti hiu qu o lng c m khng phi ch l hiu qu ni chung.

- C th chi tiu cho CNTT hn mc thng thng khi m hiu qu thng tin t c vn hn mc thng thng.

- C th chi tiu cho CNTT hn mc thng thng khi m gi tr tri thc ca nhn vin t c vn hn mc thng thng.

75 76

Nh vy, hiu qu u t CNTT trong doanh nghip cn phi o lng c v c o lng theo nhiu tiu ch, trong hiu qu thng tin v hiu qu v gi tr tri thc ca nhn vin c Paul A. Strassmann coi l hai tiu ch quan trng. iu ny hon ton ph hp vi cng thc tnh hiu qu trong l thuyt kinh t hin i, trong cng thc , gi tr tri thc va l yu t u vo, va l yu t u ra. Theo Mrten Simonsson [Simon08], doanh nghip ng i phn ln ph thuc vo CNTT, v vy vic ra quyt nh v CNTT ca doanh nghip c ngha rt quan trng.

2.1.2. Vai tr ca CNTT trong nn Kinh t tri thc

Theo Ngn hng th gii [WB2006], nn kinh t tri thc (Knowledge Economy) hay nn kinh t da trn tri thc (Knowledge-Based Economy) l nn kinh t m vic s dng tri thc l ng lc ch yu cho tng trng kinh t. Pht biu trn y khng nh vai tr ti nguyn ch yu ca tri thc trong nn kinh t. Cc quc gia c nn kinh t pht trin nht cng chnh l cc quc gia c trnh kinh t tri thc cao nht, v ngc li, cc quc gia ngho nht cng chnh l cc quc gia c trnh kinh t tri thc thp nht15.

Nn kinh t tri thc da trn bn ct tr:

(1) Mt thit ch x hi php quyn v khuyn khch kinh t (An economic incentive and institutional regime);

(2) mt lc lng lao ng c gio dc v lnh ngh (An educated and skilled labor force);

(3) mt h thng cch tn hng tri thc hiu qu (a effective innovation system);

(4) mt h tng thng tin hin i v y (a modern and adequate information infrastructure).

15 http://info.worldbank.org/etools/kam2/KAM_page5.asp.

nng cao trnh kinh t tri thc thng qua cc ct tr kinh t tri thc, cc quc gia kinh t pht trin nht th gii ch trng tng cng u t v tri thc, ch trng u t cho nghin cu-pht trin, phn mm v gio dc i hc. C th ly mt v d t bi hc Hn Quc1. Gio dc v ngun nhn lc l hai yu t ti nguyn tri thc ng gp ch cht cho s tng trng kinh t k diu ca Hn Quc trong sut bn thp nin 1960-1990. Vo nm 2004, phn ng gp ca ti nguyn tri thc cho s tng trng GDP tnh theo u ngi gp hn ba ln so vi phn ng gp ca ti nguyn c bn (bao gm vn v lao ng). Trong [WB06], Ngn hng th gii cung cp s liu v t l u t tnh theo GDP cho tri thc (u t cho nghin cu-trin khai, cho phn mm v cho gio dc i hc) v u t cho my mc v trang thit b ca cc nn kinh t pht trin nht th gii vo nm 2002 cho thy u t cho tri thc chim mt t trng cao. Hn na, trong giai on 1994-2002, xu th chung ti cc nn kinh t pht trin nht th gii l t l u t tnh theo GDP cho tri thc tng v t l u t tnh theo GDP cho my mc v trang thit b gim. Theo thng k vo nm 2010 ca T chc hp tc v pht trin kinh t (Organisation for Economic Co-operation and Development: OECD), tng u t ni a cho R&D tnh theo GDP vo nm 2008 l cao hn so vi nm 1999 a s cc quc gia (31/41) thuc t chc ny, a t l u t cho R&D trung bnh ca ton khi OECD tng t 2,16% GDP nm 1999 ln 2,28% GDP nm 200816.

Vic s dng tri thc trong nn kinh t tri thc c din ra trong cc loi hot ng l yu cu tri thc, pht sinh tri thc, ph bin tri thc v vn dng tri thc mt cch hiu qu cho tng trng kinh t. cp quc gia, tri thc l ngun ti nguyn ch yu cho tng trng kinh t, cn cp doanh nghip, tri thc l ngun to ra li th cnh tranh cho doanh nghip.

16 http://dx.doi.org/10.1787/820860264335

77 78

Hnh 2.2 V tr ca CNTT trong kinh t v m [MKG04].

Nigel Melville v cng s [MKG04] cho mt khung kinh t v m vi s tham gia ca CNTT (Hnh 2.2), trong tp trung vo v tr trong cc doanh nghip a phng.

Hnh 2.3 Khung xc nh m hnh kinh doanh (tri) v v tr ca m hnh kinh doanh trong doanh nghip (phi) [SG10].

S bn phi ca Hnh 2.3 cho thy mt gn kt b ba mt thit gia T chc kinh doanh, Chin lc kinh doanh v Cng ngh Thng tin Truyn thng (CNTT-TT) v iu ny cng khng nh

vai tr chin lc ca ICT i vi t chc. Tham gia vo b ba , Khai ph d liu v pht hin tri thc t d liu c coi l mt b phn tch cc ca CNTT.

Vai tr chin lc ca CNTT cn c th hin ch, cc h thng cung cp thng tin (ni chung) cng nh cc ng dng khai ph d liu (ni ring) ti cc doanh nghip u cn phi xut pht t yu cu kinh doanh ca doanh nghip. Nh vy, nhm tng cng tri thc ti doanh nghip, cc bi ton khai ph d liu c t ra v chng c im xut pht t nhu cu kinh doanh v phc v chin lc kinh