Text processing Campus-Booster ID : **XXXXX

www.supinfo.com Copyright © SUPINFO. All rights reserved

Awk: Generating reports

Your trainer…

Title: **Enter title or job role. Accomplishments: **What makes the presenter qualified to present this course.

Education: **List degrees if important. Publications: **Writings by the presenter on the subject of the course or presentation.

Contact: **Campus-Booster ID: presenter@supinfo.com

Presenter’s Name

Awk: Generating reports

Course objectives

n  Generate formatted reports. From any data source.

n  Apply transformation functions. On data sources.

n  Use selectors. Filter data

By completing this course, you will:

Awk: Generating reports

Course topics

n  Introduction. How it works.

n  Variables an output. Working with data.

n  Functions. Use Builtin functions and write your owns.

n  Control structures. If’s and loops.

n  Patterns. Filtering data

Course’s plan:

Awk: Generating reports

Introduction

How does it works ?

Awk: Generating reports

About Awk

n  Created in 1977 by AT&T and Bell

n  Created by Aho, Weinberger and Kernighan

n  Used to

n  Manipulate files

n  Create CGI scripts

n  Generate reports

n  Create data test and validation

Presentation:

Introduction

Synopsis Syntax of a command:

Introduction

Example:

awk [-F] [-v var=value] ‘code’ file

awk ‘{print $0}’ /etc/passwd

How it works

n  Awk uses a group of instructions called ‘Pattern‘.

n  Patterns are applied to records.

n  Records

n  Blocks of data separated by a new line or any character defined in the RS variable.

n  Fields

n  Strings in a record separated by a space or any character defined in the FS variable.

Globally:

Introduction

Fields and records

abcdef ghijklmnopqus tuvwxyz abcdef ghijklmnopqus tuvwxyz

Field Field

Introduction

Field

Record

Record

Stop-and-think Introduction

Do you have any questions ?

Stop-and-think Introduction

-FS ‘:’

-F ‘:’

-R ‘:’

-RS ‘:’

To work with /etc/passwd fields, which option and value will you use ?

Stop-and-think Introduction

-FS ‘:’

-F ‘:’

-R ‘:’

-RS ‘:’

To work with /etc/passwd fields, which option and value will you use ?

Variables and output

Pre-defined and user variables

Awk: Generating reports

About variables Variables

n  Variables

n  Pre-defined

n  User-defined

n  Types

n  Scalar

n  Tables

n  Fields

n  Records

Store and retreive values.

Pre-defined Variables and output

Variable Value

FILENAME

RS

FS

NF

FNR

Target file

Record separator

Field separator

Number of fields in a record

Number of records

Some predefined variables:

Pre-defined Variables and output

Fields and records:

Variable Value

$0

$1, $2, etc …

Whole record

Different fields

User-defined Variables example:

Variables and output

var1 = NR var2 = my_value var3 = $0 var4 = $2 var5[1]= my_value

Using variables

n  print

n  printf

Display value with print and printf:

Variables and output

awk ‘{print $0}’ file awk ‘{printf “%s - %d”, $1, $2}’ file

Stop-and-think Variables

Do you have any questions ?

Stop-and-think Variables

echo

print

printf

writeln

write

Select text-writing awk functions:

Stop-and-think Variables

echo

print

printf

writeln

write

Select text-writing awk functions:

Functions

Builtin and user-defined

Awk: Generating reports

Builtins

n  Numerical functions

n  cos(x)‏

n  sin(x)‏

n  log(x)‏

n  String functions

n  length(s)‏

n  sub(r, s, t) ‏

n  gsub(r, s, t) ‏

Some existing functions:

Functions

User-defined Syntax:

Functions

Example:

function functionname(param1, param2) { code code code

}

function hello(firstname) { print “Hello “ firstname

}

Stop-and-think Functions

Do you have any questions ?

Stop-and-think Functions

Replace every occurrence of r by s in the t string

Replace the first occurrence of r by s in the t string

Replace every occurrence of s by t in the r string

Replace every occurrence of t by s in the r string

gsub(r,s,t) does:

Stop-and-think Functions

Replace every occurrence of r by s in the t string

Replace the first occurrence of r by s in the t string

Replace every occurrence of s by t in the r string

Replace every occurrence of t by s in the r string

gsub(r,s,t) does:

Control structures

If’s and loops

Awk: Generating reports

Control code flow

n  Control structures

n  if (condition) instruction [else instruction]

n  while (condition) instruction

n  for (expression; condition; expression) instruction

n  Control commands

n  break : Loop exit

n  next : Go to the next record

n  continue : Continue loop iteration

Common control structures:

Control Structures

Examples if-then-else example:

Control Structures

while & if example:

if (var1 == 3) ‏ print “var1 = 3”

else print “var1 != 3”

i = 0 while (i < 10) {

print i i++ if (i == 5) ‏

break }

Stop-and-think Control Structures

Do you have any questions ?

Stop-and-think Control Structures

True

False

Awk has a “until” loop structure

Stop-and-think Control Structures

True

False

Awk has a “until” loop structure

Patterns

Filtering data

Awk: Generating reports

Event driven

n  You can use as a pattern

n  Regular expressions

n  BEGIN and END expressions

n  Comparison expressions

n  The three combined

Associate patterns with actions

Pattern

Syntax:

PATTERN { code code

}

BEGIN & END

n  BEGIN{action} executes the specified action before any other.

n  END{action} executes the specified action after any other.

Header and footers

Patterns

Example : awk –F':' -f script /etc/passwd BEGIN {

print “Names:\n” } {print $1 } END {

print “Total number of users: “ NR }

Other patterns

Regular expression:

Patterns

Comparison expression:

Combination:

$1 [!]~ /^[Ss]upinfo$/ { code }

$1 == 4 { code }

$1 == 2 && $4 <= 6 { code }

Stop-and-think Patterns

Do you have any questions ?

Stop-and-think Patterns

Regexp

Boolean

BEGIN

END

Which pattern will you use to produce a nice header ?

Variables

Functions Parsing files Patterns

Course summary

Separator

Awk: Generating reports

For more Awk: Generating reports

Courses Publications

Web sites

www.labo-linux.com

www.blackbeltfactory.com

Linux Technologies: Edge Computing

Conferences

FOSDEM

RMLL

Solutions Linux

If you want to go into these subjects more deeply, …

www.supinfo.com

Sed & Awk

Congratulations

You have successfully completed the SUPINFO course module n°15

Awk: Generating reports

The end

n  BEGIN & END produce nice headers/footers n  Search builtin functions before writing your owns

Awk: Generating reports