pttp & idl – external / internal file name validation

PTTP & IDL – External / Internal File Name Validation

Doug Clough

SYNERGETICS Engineered Systems

7 December 2009

www.ptagis.org

7 December 2009

Pacific States Marine Fisheries Commission

IDL – Loader Invocation

PTTP

Files are pre-processed byidl_validateFile.pl

one-by-one as IDL runs

7 December 2009


IDL – Loader Invocation

PTTP

We must detect external-internal file name

mismatch before file is presented to IDL

Files are pre-processed byidl_validateFile.pl

one-by-one as IDL runs

PROBLEM:

7 December 2009


If external and internal names don’t match …

Not only do we want to suppress file loading

• We want to prevent generation of sym-link in archival directory

• We want to return diagnostic message to sender

And what about other files in the PTTP transaction?

• Do we want to reject all of the files if one or more has name problem?

• Do we want to reject only the specific files that have problems?

In either case, we need to take action upstream of PTTP_Ack::genEvents()

Have a look at PTTP_Req::analyze()

• Loops through all files in the TX

• Calls FDVL_ArchiveTree::makeArchiveLink()– Insert code before this call to compare internal and external names– If the names match, create the link and continue as presently coded– Otherwise, invoke new method on PTTP_Ack to set OPER value for this file to REFUSED– This will suppress generation of an invocation event for the file

7 December 2009


PTTP Detail

Process flow …

1) watchFtp launches processPittagFtp

2) PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req

3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event

4) PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

Symbolic link in

archive directory

is created first …

7 December 2009


PTTP_Req::analyze()

sub analyze { my ( $obj ) = @_;

$TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;

( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));

foreach $FileName ( @DataFiles ) {

# NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze.

$FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");

if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) {

# Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1;

# Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent. FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');

$HasFilesToAnalyze = 'True'; } }

7 December 2009


FDVL_ArchiveTree::makeArchiveLink()

sub makeArchiveLink { my ( $PhysicalPath , $ArchivalPath , $FullSpec , $CmdString ); my ( $SourceDir , $FileName , $FileType # INT, MUL, or TAG (INT and MUL give same result) ) = @_;

$PhysicalPath = "${SourceDir}/${FileName}"; $ArchivalPath = &sayAbsolutePath($FileName,$FileType); $FullSpec = "${ArchivalPath}/${FileName}";

unless ( -f $FullSpec ) {

# Create directories if they don't exist ... &checkDirs($FileName,$FileType);

$CmdString = "ln -s $PhysicalPath $FullSpec";

system("$CmdString");

} else {

print "makeArchiveLink ...\n" if ( $DEBUG ); print " Already exists: $FullSpec\n" if ( $DEBUG ); $DUMMY = <STDIN> if ( $DEBUG ); }

}

7 December 2009


PTTP Detail

Process flow …





How are files presented

for loading?

7 December 2009


PTTP Detail

Process flow …



3) PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event(s)


For ‘Invocation’ event PTTP_Dispatcher …

• Invokes handler

• Handler reads possible Input file

• Handler performs action and queues‘Completion’ event

File Type Handler

INTERROGATION PTTP_intIDL

MULTIMON PTTP_mulIDL

TAGGING PTTP_FDVL

run_loader($evt){<copies $FileName to $INTERROGATION_DATA>}

One per file

7 December 2009


PTTP_intIDL::runLoader()

sub runLoader { my ( $TX_ID , $F_NAME , $F_TYPE , $RetVal , $IDL_Name , $CmdString , $DEBUG );

$DEBUG = 0; #$DEBUG = 1;

my ( $r_evt ) = @_;

$TX_ID = $$r_evt->get("PTTP_Event.TX_ID"); $F_NAME = $$r_evt->get("PTTP_Event.EventData.F_NAME"); $F_TYPE = $$r_evt->get("PTTP_Event.EventData.F_TYPE");

# Assemble file name for use by Interrogation Data Loader (IDL) ... $IDL_Name = $F_NAME . ":" . $TX_ID . ":" . $F_TYPE;

# Make file available for processing by IDL ... $CmdString = "cp " . "$PTTP_SERVER_PERSIST_DIR/$TX_ID/$F_NAME" . " " . "$PTTP_INT_DATA/$IDL_Name"; system("$CmdString");

}

7 December 2009


PTTP Detail

Process flow …





What we want:

Only do this if external

and internal file

names match!

7 December 2009


PTTP Detail

Process flow …





In other words:

Block event generation

if names don’t match

7 December 2009










Have a look at PTTP_Req::analyze()



7 December 2009










Have another look at PTTP_Req::analyze()



7 December 2009


PTTP_Req::analyze()

sub analyze { my ( $obj ) = @_;

$TX_ID = $obj->get("PTTP_Req.TX_ID"); $TX_Dir = $PTTP_SERVER_PERSIST_DIR . "/" . $TX_ID;

( @DataFiles ) = split(/\n/,$obj->listKeys("PTTP_Req.FILES"));

foreach $FileName ( @DataFiles ) {

# NOTE: Assume that all files of type 'INT' and 'MUL' # will be analyzed by TASS and SBCA. Leave it to # TASS_Monitor and SBCA_Monitor to decide which # files to analyze.

$FileType = $obj->get("PTTP_Req.FILES.$F_Name.F_TYPE"); $ReqOper = $obj->get("PTTP_Req.FILES.$F_Name.REQUEST.OPER");

if ( ( $FileType =~ /^INT.*$|^MUL.*$/i ) && ( $ReqOper =~ /LOAD/i ) ) {

# Capture $SiteCode for use below ... $FileName =~ /^\s*(...).*$/; $SiteCode = $1;

# Unless the file is already present in the final FDVL archive # location, create a symbolic link there pointing to the file in # $TX_Dir. FileType not important here; INT and MUL are equivalent. FDVL_ArchiveTree::makeArchiveLink($TX_Dir,$FileName,'INT');

$HasFilesToAnalyze = 'True'; } }

7 December 2009













7 December 2009


PTTP_Ack.pm

#------------------------------------------------------------------------------------------# I N I T I A L I Z A T I O N A N D D E C L A R A T I O N S#------------------------------------------------------------------------------------------

# Includes ...use PTTP_Message;use PTTP_Event;use HashAssemblage;@ISA = ("PTTP_Message","HashAssemblage");

# Initialize operation precedence ...@OpSeq = ( 'OPER' # File LOAD or CORRECTION , 'STAT' # File record-type statistics , 'HIST' # History of OPERations on file , 'TEST' # Move files to "test archive" - do NOT load );

# Initialize list of status values for each operation that# indicate completion or failure (i.e. final state) ...%FiniStat = ( 'OPER' => 'LOADED|REFUSED|INVALID|NET_DUPS|NET_ANOMALIES|SYS_FAIL|AUTH_FAIL|AGG_FAIL' , 'STAT' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'HIST' => '\||Not Performed|NON_EXIST|SYS_FAIL|AUTH_FAIL' , 'TEST' => 'ARCHIVED|FAILED|SYS_FAIL|AUTH_FAIL' );

7 December 2009


PTTP_Ack::genEvents()

# Generate an instance of PTTP_Event to initiate processing of first# request for each file named in the A_AST "To Do" list ...sub genEvents { my ( $FileList , @Files , $File , $FileType , $F_Name , $NextOp , $OpVal , $TX_ID , $evt , $EventData , $CanonicalDNA , $DBG_File );

my ( $ack # Reference to PTTP_Ack object instance ) = @_;

# Obtain list of files named in the "To Do" list ... $FileList = $ack->opFiles();

@Files = split(/\n/,$FileList);

# Generate an event for each file with un-fulfilled requests ... # NOTE: At this point, all of the files should qualify. foreach $File ( @Files ) {

$ack->genNextEvent($File);

}}

7 December 2009


PTTP_Ack::genNextEvent()

# Generate an instance of PTTP_Event to initiate processing of the# next request for the specified file ...sub genNextEvent { my ( $ack # Reference to PTTP_Ack object instance , $File ) = @_;

# Protect embedded '.' ... $F_Name = $File; $F_Name =~ s/\./\.\./g;

$NextOp = $ack->nextOp($File); $TX_ID = $ack->get("PTTP_Ack.TX_ID"); $OpVal = $ack->get("PTTP_Ack.FILES.$F_Name.REQUEST.$NextOp"); $FileType = $ack->get("PTTP_Ack.FILES.$F_Name.F_TYPE"); unless ( $NextOp =~ /None/i ) { $evt = new PTTP_Event(); $evt->set("PTTP_Event.TX_ID='$TX_ID'"); $evt->set("PTTP_Event.Source='PTTP_Client'"); $evt->set("PTTP_Event.Target='PTTP_Server'");

$EventData = "PTTP_Event\n" . " EventData\n" . " F_NAME | " . $File . "\n" . " F_TYPE | " . $FileType . "\n" . " " . $NextOp . " | " . $OpVal . "\n" ;

$evt->growStructure($EventData); $evt->persist(); } }

7 December 2009


PTTP_Ack::nextOp()

# Returns the 'next operation' in sequence for a given FileName.# If all operations are complete, returns 'None'.sub nextOp { my ( $F_Name , $OpName , $ReqString , @Requests , $Request , $ReqStat , $FinalStat );

my ( $obj , $FileName ) = @_;

$F_Name = $FileName; $F_Name =~ s/\./\.\./g; # Protect embedded '.' $ReqString = $obj->listKeys("PTTP_Ack.FILES.$F_Name.REQUEST"); ( @Requests ) = split(/\n/,$ReqString);

foreach $OpName ( @OpSeq ) { foreach $Request ( @Requests ) { if ( $OpName =~ /$Request/i ) { $ReqStat = $obj->get("PTTP_Ack.FILES.$F_Name.REQUEST.$OpName"); $FinalStat = $FiniStat{$OpName}; unless ( $ReqStat =~ /$FinalStat/ ) {

# Generate return value ... return $OpName; } } } }

# Generate return value ... return 'None';}

REFUSED is a FinalState

7 December 2009













PTTP & IDL – External / Internal File Name Validation

Doug Clough

SYNERGETICS Engineered Systems

7 December 2009

www.ptagis.org

That’s all, folks!

pttp & idl – external / internal file name validation

Documents