heritrix release notes · 6. release 1.12.1 - 2007-05-06 release 1.12.1 is primarily a bug-fix...

Heritrix Release NotesInternet Archive

Table of Contents1. Release 1.14.4 - 2010-04-30 ...................................................................................... 22. Release 1.14.3 - 2009-03-02 ...................................................................................... 23. Release 1.14.2 - 2008-08-06 ...................................................................................... 24. Release 1.14.1 - 2008-08-06 ...................................................................................... 25. Release 1.14.0 - 2008-04-27 ...................................................................................... 26. Release 1.12.1 - 2007-05-06 ...................................................................................... 3

6.1. Contributors ................................................................................................. 36.2. All tracked changes ....................................................................................... 3

7. Release 1.12.0 - 2007-03-16 ...................................................................................... 37.1. Contributors ................................................................................................. 37.2. Notes .......................................................................................................... 37.3. Known Limitations/Issues ............................................................................... 37.4. Changes ...................................................................................................... 4

8. Release 1.10.2 - 2007-01-15 ...................................................................................... 48.1. Contributors ................................................................................................. 58.2. Changes ...................................................................................................... 5

9. Release 1.10.1 - 2006-09-27 ...................................................................................... 89.1. Changes ...................................................................................................... 8

10. Release 1.10.0 - 2006-09-11 ..................................................................................... 810.1. Contributors ............................................................................................... 810.2. Known Limitations/Issues ............................................................................. 910.3. Pre-1.10.0 checkpoints .................................................................................. 910.4. Changes ..................................................................................................... 9

11. Release 1.8.0 - 2006-05-05 .................................................................................... 2211.1. Known Limitations/Issues ........................................................................... 2211.2. Changes ................................................................................................... 22



14. Release 1.2.0 - 2004-11-16 .................................................................................... 6914.1. Known Limitations .................................................................................... 6914.2. Changes ................................................................................................... 70

15. Release 1.0.4 - 2004-09-22 .................................................................................... 8015.1. Changes ................................................................................................... 80

16. Release 1.0.2 - 2004-09-14 .................................................................................... 8016.1. Changes ................................................................................................... 81

17. Release 1.0.0 - 2004-08-06 .................................................................................... 8117.1. Known Limitations .................................................................................... 8217.2. Changes ................................................................................................... 83

18. Release 0.10.0 - 2004-06-04 ................................................................................... 8618.1. Changes ................................................................................................... 86

19. Release 0.10.0 - 2004-06-04 ................................................................................... 9219.1. Changes ................................................................................................... 92

20. Release 0.8.0 - 2004-05-18 .................................................................................... 92

1

20.1. Synopsis .................................................................................................. 9220.2. Changes ................................................................................................... 92

21. Release 0.6.0 - 2004-03-25 .................................................................................... 9821.1. Changes ................................................................................................... 98

22. Release 0.4.1 - 2004-06-04 .................................................................................. 10222.1. Changes ................................................................................................. 102

23. Release 0.4.0 - 2004-02-10 .................................................................................. 10223.1. Changes ................................................................................................. 102

24. Release 0.2.0 - 2004-01-05 .................................................................................. 10325. Release 0.1.0 - 2003-12-31 .................................................................................. 103

1. Release 1.14.4 - 2010-04-30Release 1.14.4 is a bugfix release with 44 tracked fixes and small improvements.

Other details on this release, including a list of all tracked issues addressed, are available at: ReleaseNotes - Heritrix 1.14.4 at project wiki[https://webarchive.jira.com/wiki/display/Heritrix/Release+Notes+-+1.14.4]

2. Release 1.14.3 - 2009-03-02Release 1.14.3 is a bugfix release with 14 tracked fixes and small improvements.

Other details on this release, including a list of all tracked issues addressed, are available at: ReleaseNotes - Heritrix 1.14.3 at project wiki[http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.3]

3. Release 1.14.2 - 2008-08-06Release 1.14.2 is a bugfix release with 6 tracked fixes.


4. Release 1.14.1 - 2008-08-06Release 1.14.1 is a bugfix release with 25 tracked fixes and small requested/contributed features.


5. Release 1.14.0 - 2008-04-27Release 1.14.0 adds a number of small features to the Heritrix 1.x line, most notably upgrading supportfor the WARC archived-web-content format to version 0.17 (ISO Committee Draft). This release alsoincludes 41 bug fixes or other incremental improvements.


Heritrix Release Notes

2

https://webarchive.jira.com/wiki/display/Heritrix/Release+Notes+-+1.14.4

https://webarchive.jira.com/wiki/display/Heritrix/Release+Notes+-+1.14.4

http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.3








6. Release 1.12.1 - 2007-05-06Release 1.12.1 is primarily a bug-fix release. A total of 21 bugs have been addressed, including an ob-ject-retention issue when running multiple jobs in sequence, a failure to enforce fetch timeouts on cer-tain malformed server responses, and decide-rule overrides disappearing when basing jobs on earlierjobs/profiles. There are also several enhancements, such as reporting the volume of data evaluated butnot transferred/stored when using duplicate-reduction features.

6.1. ContributorsAside from the usual suspects [http://crawler.archive.org/team-list.html], the following people contrib-uted to this release:

• Ahmed Ghouzia

6.2. All tracked changesA dynamic list of all tracked changes marked as fixed in 1.12.1 is available at: Issues with 'Fix Version'1.12.1.[http://webteam.archive.org/confluence/display/Heritrix/Issues+with+%27Fix+Version%27+1.12.1]

7. Release 1.12.0 - 2007-03-16Release 1.12.0 is the first of several planned releases enhancing Heritrix with "smart crawler" function-ality. In this release, the theme has been offering new options to reduce the amount of duplicate contentcrawled and stored when recrawling sites at regular intervals. A number of other enhancements and bugfixes are also included.

7.1. ContributorsAside from the usual suspects [http://crawler.archive.org/team-list.html], the following contributed tothis release:

• Oskar Grenholm

• Doug Judd

7.2. NotesWith this release, Heritrix project issue-tracking has moved from Sourceforge to a JIRA-based system athttp://webteam.archive.org/jira/browse/HER [http://webteam.archive.org/jira/browse/HER].

Those using Heritrix in a Hadoop environment may be interested in Doug Judd's HDFSWriterProcessor[http://www.zvents.com/labs/hdfs_writer_processor], for storing crawled content directly into HDFS,the Hadoop Distributed FileSystem [http://lucene.apache.org/hadoop/].

7.3. Known Limitations/Issues

7.3.1. java.io.IOException: No locks available


3

http://crawler.archive.org/team-list.html

http://webteam.archive.org/confluence/display/Heritrix/Issues+with+%27Fix+Version%27+1.12.1



http://webteam.archive.org/jira/browse/HER

http://www.zvents.com/labs/hdfs_writer_processor

http://lucene.apache.org/hadoop/

See Section 11.1.1, “java.io.IOException: No locks available” in 1.8.0 Release Notes.

7.3.2. Older Checkpoints

Checkpoints from earlier versions are generally not supported for resume in later versions.

7.3.3. Older configurations (order.xml, etc.)

Crawler configuration files from jobs in previous versions may work in 1.12.0, though missing new set-tings will be set to their default values, and obsolete old settings will generate log warnings. Re-creatingconfigurations from defaults or hand-editting to match newer files is recommended.

7.4. Changes

7.4.1. Duplication reduction features

A collection of Processors, including the FetchHistoryProcessor, PersistProcessor, and its subclasses,may be used together with new options on the FetchHTTP and writer processors to carry informationforward between crawls and collect less duplicate content on later recrawls. The project wiki featuresnotes on using the new duplication-reduction functionality[http://webteam.archive.org/confluence/display/Heritrix/Feature+Notes+-+1.12.0].

7.4.2. DecideRules have replaced Filters on Processors

All Processors which used internal Filters for differentially acting on URIs now use DecideRules in-stead. In those cases where a DecideRule replacement for a Filter is not yet available, a legacy Filter canbe wrapped in a FilterDecideRule to preserve prior functionality. In a future release, all Filters will beremoved in favor of equivalent DecideRules.

7.4.3. WARC

ExperimentalWARCWriter has been updated to match proposed WARC version "WARC/0.12"(revision H1.12-RC1)[http://archive-access.svn.sourceforge.net/viewvc/*checkout*/archive-access/branches/gjm_warc_0_12/warc/warc_file_format.html]. The implementation as of Heritrix 1.10.x remains for reference asorg.archive.io.warc.v10.ExperimentalV10WARCWriterProcessor. The WARC format remains underdiscussion.

7.4.4. Kw3WriterProcessor

Oskar Grenholm of the Swedish National Library has contributed a module that writes the results of suc-cessful fetches to files on disk. These files are MIME-files of the type used by the Swedish NationalLibrary's Kulturarw3 web harvesting [http://www.kb.se/kw3/].

7.4.5. All tracked changes

A dynamic list of all tracked changes marked as fixed in 1.12.0 is available at: Issues with 'Fix Version'1.12.0.[http://webteam.archive.org/confluence/display/Heritrix/Issues+with+%27Fix+Version%27+1.12.0]

8. Release 1.10.2 - 2007-01-15This is primarily a bug-fix release, with a couple of new features, provided before a number of signific-ant changes to the Heritrix project that will require developer and crawl operator adjustments. Post-


4

http://webteam.archive.org/confluence/display/Heritrix/Feature+Notes+-+1.12.0

http://archive-access.svn.sourceforge.net/viewvc/*checkout*/archive-access/branches/gjm_warc_0_12/warc/warc_file_format.html

http://archive-access.svn.sourceforge.net/viewvc/*checkout*/archive-access/branches/gjm_warc_0_12/warc/warc_file_format.html

http://www.kb.se/kw3/



1.10.2, Heritrix source code control, issue tracking, and build process will migrate to new systems. Also,updates to core classes, especially with regard to the settings architecture, will noticeably break back-ward compatibility with 1.10.2 and prior crawler settings files and formats.

8.1. Contributors

• Olaf Freyer

• Max Schöfmann

8.2. Changes

8.2.1. Jericho HTML Extractor

Olaf Freyer has contributed an HTML Extractor named JerichoExtractorHTML based on the JerichoHTML Parser. Following is a quote from the JerichoExtractorHTML class comment describing how thenew Extractor differs from ExtractorHTML, its advantages and downsides: “ This extractor extends Ex-tractorHTML and mimics its workflow - but has some substantial differences when it comes to internalimplementation. Instead of heavily relying upon java regular expressions it uses a real html parser lib-rary - namely Jericho HTML Parser (http://jerichohtml.sourceforge.net). Using this parser it can betterhandle broken html (i.e. missing quotes) and also offer improved extraction of HTML form URLs (notonly extract the action of a form, but also its default values). Unfortunately this parser also has one ma-jor drawback - it has to read the whole document into memory for parsing, thus has an inherent OOMErisk. This OOME risk can be reduced/eleminated by limiting the size of documents to be parsed (i.e. us-ing NotExceedsDocumentLengthTresholdDecideRule). Also note that this extractor seems to have alower overall memory consumption compared to ExtractorHTML. (still to be confirmed on a largerscale crawl) ”

Table 1. All Tracked Changes

ID Type Summary Open Date By Filer

913002[https://sourceforge.net/tracker/index.php?func=de-tail&aid=913002&group_id=73833&atid=539102]

Add Make Extract-orHTML ag-gressivenessconfigurable

2004-03-09 gojomo gojomo


Add [Contrib]JerichoExtract-orHTML

2006-10-09 nobody pandae

1633458 Add [arcreader] Sup- 2007-01-11 stack stack


5

https://sourceforge.net/tracker/index.php?func=detail&aid=913002&group_id=73833&atid=539102




[https://sourceforge.net/tracker/index.php?func=de-tail&aid=1573708&group_id=73833&atid=539102]

port for s3 andstreaming im-provements


Fix filehandle leak:ReplayInput-Stream/BufferedSeek-InputStream

2007-01-05 karl-ia gojomo


Fix "failed get ofreplay" in Ex-tractorHTML...usu: UTF-16BE



Fix Fix DNSJavaissues(memory)



Fix ExtractorDOCwants wholedoc in memory

2006-04-26 paul_jack gojomo


Fix Do not allowhttp:/ and https:/urls

2006-12-19 stack-sf stack-sf

1596176[https://sourcefo

Fix NotMatchesLis-tRegExpDeci-



6








rge.net/tracker/index.php?func=de-tail&aid=1596176&group_id=73833&atid=539099]

deRule extendswrong class


Fix NPE in quotaE-nfor-cer.checkQuotas

2006-11-09 nobody svc


Fix [PATCH]Webapp doesn'tfind profilesand ignoresjobsdir

2006-10-30 nobody nobody


Fix SURTs for IP-address URIsunhelpful



Fix NPE in Fetch-HT-TP.saveCookies

2006-06-06 gojomo stack-sf


Fix Useragent com-pare because ofcase in Robot-sExclusion-Policy



7






9. Release 1.10.1 - 2006-09-27Bug fixes.

9.1. Changes



1566041[http://sourceforge.net/tracker/index.php?func=detail&aid=1566041&group_id=73833&atid=539099]

Fix In 1.10.0, ARCserialno starts at1 rather than 0



Fix JNDI registra-tion fails in1.10.0



Fix StripWWWN-Rule does notwork as advert-ised



Fix ARCWriterconstructor hadits public accessremoved


10. Release 1.10.0 - 2006-09-11Release 1.10.0 adds new configuration options, experimental new protocol and format support, and lotsof fixes. 43 tracked bugs have been fixed and 35 feature requests added.

Release 1.10.0 requires JDK 1.5.x ("Java 5") Java facilities.

10.1. Contributors


8

http://sourceforge.net/tracker/index.php?func=detail&aid=1566041&group_id=73833&atid=539099




Aside from the usual suspects [http://crawler.archive.org/team-list.html], the following contributed tothis release:

• Eric C Jensen

• Olaf Freyer

• Karl Wright (of MetaCarta)

• Frank McCown (of Old Dominion University)

• Max Schöfmann

• Søren Vejrup Carlsen (of Royal Library, Denmark)



See Section 11.1.1, “java.io.IOException: No locks available” in 1.8.0 Release Notes.

10.3. Pre-1.10.0 checkpointsFor sure 1.8.0 checkpoints will not be recoverable with 1.10.0.

10.4. Changes

10.4.1. No default login/password for web UI and JMX

The old default login of 'admin' and password of 'letmein' for access to the crawler web UI (and JMXagent control) have been eliminated. It is now necessary to specify an access username and password tostart Heritrix. This may be done with the -a or --admin command-line argument or via the system prop-erty 'heritrix.cmdline.admin'. (These each take a colon-separated username and password, like 'user-name:password'.)

10.4.2. Web UI binds to localhost only by default

Previously, the Jetty web server that runs the Heritrix web UI listened on all available network inter-faces. In 1.10.0, Jetty will only bind to localhost by default. The -b or --bind command-line argumentcan be used to specify a different interface or list of interfaces to bind to instead. You may specify "-b /"to get the old behavior -- binding on all interfaces -- but only take this step after reading section 2.3 ofthe User Manual, "Security Considerations".

10.4.3. QuotaEnforcer 'force-retire' option

The optional QuotaEnforcer processor has a new setting, 'force-retire', which is by default 'true', andchanges the default behavior of QuotaEnforcer. Previously, when a URI was noted as being over-quota,it would be marked with a special over-quota failure code which caused it to complete processing as anerror. As a result, all over-quota URIs would quickly be finished as errors and appear in the crawl.log,but there would be no opportunity to raise the quota and continue crawling.

The new default behavior instead marks the URI with a directive requesting its frontier queue be retired.If the frontier supports this directive, the URI will be returned to its queue as if never tried, and the


9


whole queue retired from active crawling. This offers the opportunity to raise the quota and continuecrawling the URI and others of its queue. (All settings changes cause all retired queues to be reevalu-ated.) However, the over-quota URIs will not appear as errors in the crawl.log.

If the old behavior is preferred, set 'force-retire' to 'false'.

10.4.4. URL canonicalization changes

In 1.10.0, URL canonicalization has changed in two ways. First, the stripping of sessionids has im-proved [See Stripping sessionid can leave behind doubled ampersands[http://sourceforge.net/tracker/index.php?func=detail&aid=1550797&group_id=73833&atid=539099]].Previous, if the sessionid was in the middle of a query string bookended by other query parameters, ca-nonicalization would leave behind the encasing ampersands: E.g. If the URL ht-tp://a.com/?a=1&sid=00000000000000000000000000000000&b=1 was passed throughcanonicalization, the result would be: http://a.com/?a=1&&b1. This has been fixed so that theresult will now be: http://a.com/?a=1&b1.

The second change, [1550805] Add stripping of coldfusion sessionids[http://sourceforge.net/tracker/index.php?func=detail&aid=1550805&group_id=73833&atid=539102],adds the new coldfusion sessionid stripper to the list of default canonicalization rules.

We bring your attention to these seemingly minor changes because for those of you running regularcrawls, with both of the above changes in place, depending on the type of crawl, there should be a reduc-tion in overall the number of (duplicate) pages crawled.

10.4.5. WARC

This release includes experimental WARC readers and writers. Be warned that both code and specifica-tion are not yet final and so are both subject to change with no guarantees of backward compatibility: i.e.newer readers may not be able to read WARCs written with older writers. See the org.archive.io.warc[/apidocs/org/archive/io/warc/package-summary.html] package documentation for more on the currentstate of code including documentation of initial version of Arc2Warc and Warc2Arc tools.

10.4.6. FTP

This release also include experimental support for FTP. This support is disabled by the default heritrixconfiguration. See the User Guide for information on how to enable FTP.




Add ExperimentalWARC Readersand Writers


1494491[http://sourceforge.net/tracker/index.php?func=detail&aid=1494

Add path/role-sensitiverobots (eg ig-nore for inlineimages/css)



10



/apidocs/org/archive/io/warc/package-summary.html




491&group_id=73833&atid=539102]


Add 'Implied' URIextractor (eg,YouTube)



Add Add experi-mentalWarc2Arc andArc2Warcscripts



Add Secure adminUI: Bind cmd-line argument

2006-08-25 karl-ia stack-sf


Add remove defaultadmin user-name/password



Add hash-basedCrawlMapper



Add force reread ofdisk settings(for out-of-JVM/bulkchanges)


1534280[http://sourceforge.net/tracker/in

Add scriptable(beanshell) Pro-cessor, Deci-



11









dex.php?func=detail&aid=1534280&group_id=73833&atid=539102]

deRule options


Add CrawlMapperskip mapping'E'mbeds (etc)



Add keep over-limit(-500X) URIsin queues (don't'finish/log)



Add [arcreader]Fetch recordsand iterate re-mote ARCs



Add favicon.ico forheritrix web ui



Add [contrib] AddBigMapFact-ory.getSynchronizedBigMap

2005-05-27 gojomo ck-heritrix


Add broader rotation/ wider 'front-line' frontierqueue option


1092496 Add crawl.log 2004-12-28 stack-sf gojomo


12









[http://sourceforge.net/tracker/index.php?func=detail&aid=1092496&group_id=73833&atid=539102]

should havehash of DNS re-cords


Add FTP fetching 2004-08-09 karl-ia gojomo


Add Add strippingof coldfusionsessionids --add to defaultlis



Add [contrib] patchto allow settinglocal IP to bindfetch from

2006-08-26 stack-sf ecjensen


Add [contrib] allowto specify al-ternative conflocation

2006-08-24 stack-sf pandae


Add [contrib] Con-tentLengthFilter

2006-08-24 stack-sf pandae


Add Add check-pointing selftest



13









Add Add creation/deletion of Her-itrix instancesto UI



Add [contrib] En-hanced UI seedand crawl re-ports



Add Should supportdepth-firstsearch priorityscheduling(patch

2006-07-15 stack-sf ecjensen


Add Improved hand-ling when al-loted runtime isexceeded

2006-07-07 kristinn_sig kristinn_sig


Add (contrib)Provide Win-dows batch fileversion ofscripts

2006-06-29 nobody ecjensen


Add [contrib] HaveHeritrix UI bindto localhostonly


1505111[http://sourceforge.net/tracker/index.php?func=detail&aid=1505111&group_id=73833&atid=53

Add Make deciding-default profilethe default pro-file



14









9102]


Add Move to java5.0/1.5.0

2006-05-15 nobody stack-sf


Add [contrib] Throt-tling on a per-document basis



Add change user-name/passwordafter launch



Add Show old crawlreports in UI(Was: Reportson finished...)



Add Fix API to al-low ARCWriterreplacement



Fix proxying of ht-tps gives errors/garbage/laterproblems



Fix override ofuser-agents andmasquerade notworking

2006-08-03 karl-ia ia_igor


15









082&group_id=73833&atid=539099]


Fix multiple usageof same arc idnumber withinsame crawl



Fix Checkpointingis broken (Parts1 and 2)



Fix incorrectresolving relat-ive links fromflash files (swf)



Fix CSS keywordsare case senset-ive in extraction

2006-06-21 gojomo cathcart


Fix Contain Http-Client HttpPars-er'sOutOfMemory-Error risk



Fix HTMLExtractorand application/xhtml+xmltype?



Fix Archive fileserialnumber isalways 1 after



16










checkpoint


Fix Don't resumecrawl aftercheckpoint ifstate is 'pausing'



Fix adjust promin-ence of in-stance/identifierinfo/tab



Fix FetchDNS IO-Exception:Stream closed



Fix HeritrixProto-colSocketFact-ory synchroniz-ation causesdelays

2006-08-11 stack-sf gojomo


Fix don't insist onrobots.txt if itneed not behonored



Fix OnDomainsDe-cideRule notworking as ex-pected


1532665 Fix AddRedirect- 2006-08-01 gojomo gojomo


17










FromRoot-ServerToScopenot working asexpected


Fix IPQueueAs-signmentPolicybroken bymethod signa-ture mismatch



Fix heritrix fails tosave accept-headers in anoverride

2006-06-29 karl-ia magin-ia


Fix NoOnDomains-DecideRule/NotOnHostsDe-cideRule super-class wrong



Fix CachedBd-bMap.keySetinefficient orbroken



Fix ARCRead-er#read(byte [],off, len) brokefor non-null off-set



Fix ARC problemcausing .invalidsuffix needsbetter reporting

2005-04-25 paul_jack gojomo


18









Fix NPE at Crawl-StateUp-dater.java:70 ht-tp:/robots.txt



Fix doc securityconsiderations



Fix [jdk1.6] Com-plex-Type#toStringinfinite loop



Fix Concurrent-Modifica-tionExceptionin web UI fron-tier report



Fix LinksScoperscope-em-bedded-links in-consistent/con-fusing



Fix UURIFactory '//' collapsingovereager



Fix queued countwrong with re-tired queues;crawl doesn'tend



19









9099]


Fix ARCWriterPoolnot fair tothreads



Fix regex for mid-fetch filter notbeing stored incrawl order

2005-12-12 gojomo nobody


Fix Stripping ses-sionid can leavebehind doubledampersands



Fix excessiveWakeTask maybe scheduled



Fix Remove Mir-rorJNDI. ItsGPL



Fix [extractorhtml]Passes throughentity-en-codings



Fix Job's crawl re-port link pro-duces report fordifferent job

2006-07-03 nobody fmccown


20









354&group_id=73833&atid=539099]


Fix Browsers toler-ate newlines inURLs, Heritrixdoesn't



Fix Values fromdropdown get-ting tacked onfor next hit.

2006-06-16 stack-sf nobody


Fix [jmx] Add re-bind to JNDI



Fix hangs withqueued docu-ments not beingassigned toqueues



Fix httpclient list ofproto-factoriesis static



Fix Non-serializableclass AR-CReader con-tains Exception

2006-05-01 stack-sf lars_clausen


Fix escapeJavaS-cript should es-cape HTML

2006-04-13 paul_jack pandae


21










problem charac-ters

11. Release 1.8.0 - 2006-05-05Release 1.8.0 adds a number of minor improvements and fixes. Most notably, checkpointing can now beachieved with a single command (with the requisite pause/resume done automatically), and all URIsfetched may be tagged with the original seed URI from which they were discovered. (This source URIinformation is both in the crawl.log and a new 'source-report.txt' report available among the disk file re-ports.)

We expect release 1.8.0 to be the last release officially supported on JDK 1.4.x ("Java 2") Java; futurereleases will require JDK 1.5.x ("Java 5") Java facilities.



BDB-JE will complain 'No locks available' when crawler is being built/run on an NFS mount. Work-around is to locate the 'state' directory on a non-NFS-mounted volume.

11.1.2. "Channel closed, may be due to thread interrupt"

An error with this message has been observed intermittently when running on the Sun Java 6("mustang") beta JVM ("-beta2-b81"). A forthcoming fix from Sleepycat for BDB-JE may be necessaryto resolve this issue.

11.2. Changes

11.2.1. Progress Statistics Log

The format of progress statistics' state-change log messages have been modified. State-change messagesnow have a tail that adds some context explaining why we're pausing, etc. Note, we will be adding ori-ginator of the status-change event to the progress statistics log post 1.8.0 -- i.e. whether event came ofJMX or via the UI -- so be prepared for more progress log changes.

11.2.2. Checkpoints

Now when you ask to checkpoint a running crawl, it will manage for you the pause, checkpoint, and re-sume cycle (If paused when checkpoint is invoked, the crawler will be set back into a paused state uponcheckpoint completion).

Checkpoints made with 1.6.0 software cannot be recovered with 1.8.0 software. Core classes such asCrawlController have changed so their serialized representation as part of a checkpoint has also changed(We have not done the work to deserialize earlier versions of core classes serialized as part of a check-point).


22




Fix upping totalbudget doesn'tupdate/unretirequeues



Fix BDB Adler32gc-lock OOMErisk



Fix [jmx] Makedownloadeddata count haveconstant units



Fix refactor/com-pact QuotaEn-forcer code



Fix crawl report/hosts-reportstats leave outrobots.txt



Fix Failed deregis-tration of con-tainer with jndi



Fix When multipleinstances,there's always a



23










runt in the litter


Fix JMX get alertby indexbroken.



Fix Corruptjob.state filesobstruct crawlresumption



Fix stop alerts 'linein seed file ig-nored' formixed seed/surt

2006-03-02 gojomo ia_igor


Fix IllegalArgu-mentExceptionadding tosource host re-port



Fix make_reports.ploutdated,broken



Fix OnHostsDeci-deRule/OnDomainsDe-cideRule notadding seedSURTs


1475638 Fix Robots.txt ig- 2006-04-24 gojomo stack-sf


24










nored if206/203 StatusCode


Fix crawl.log en-tires do not re-flect 'no spaceleft' error



Fix Extract-orHTML/Ex-tractorJS 'hang'on many-backslash input



Fix ExtractorCSSdoes not resolverelative URIsagainst BASE



Fix ExtractorJSNPE doingspeculative ex-traction



Add [arcreader]Fetch recordsand iterate re-mote ARCs



Add [jmx] Addname of heritrix'host' as att



25









Add replaceutil.concurrentwithjava.util.concurrent



Add [jmx] Regular-ize crawl endstate messages



Add JmxUtils.getOpenType() musthandle Doubles



Add [jmx] progressstatistics as no-tification



Add [contrib]Preselector AT-TR_ALLOW_BY_REGEXP



Add 'total' bytes/fetches quotaoptions inQuotaEnforcer



Add Carry forward(& log) 'origin-ating URL/seed'for all URLs



26









9102]


Add Add destroy toJMX API



Add New "seedsource report"of # of URLsper host persource



Add improve surtdocs; esp 'surts-source-file' syn-tax

2006-02-21 nobody gojomo


Add unattendedcheckpointing


12. Release 1.6.0 - 2005-12-01Release 1.6.0 offers improved remote control and monitoring via JMX, a crawl-checkpointing facility,and experimental support for bloom filter already-included testing, partitioning a crawl across multipleindependent crawlers, and per-host/domain/queue-grouping collection quotas. Performance and stabilityin large crawls is also improved. Among tracked issues, it includes 39 requested enhancements and fixes96 reported bugs.



BDB will complain 'No locks available' when crawler is being built/run on an NFS mount. Workaroundis not run on an NFS-mounted volume.

12.1.2. OutOfMemoryError in 64bit JVMs

BDB 2.0.90 can overgrow its intended cache size due to a misestimation of instance sizes under 64bit


27





Java VMs, which may be a major contributor to early Heritrix OutOfMemoryError problems on 64bitsystems. A workaround is to cut the assigned percentage by 1/3 to 1/2. For example, change the 'bdb-cache-percent' setting to '40' or '30' (instead of the default 60% when no value is set here).

12.2. Changes

12.2.1. Postselector

The Postselector has been refactored out of existence. Its responsibilities have been parcelled out to twonew Processors: LinksScoper and FrontierScheduler. LinksScoper is responsible for scope checking ofextracted links. FrontierScheduler does the scheduling of URIs with the Frontier.

This change was done to allow introduction of processors between scope checking and Frontier schedul-ing steps.

Because of this change, order files from 1.4.0 Heritrix or before will need to be updated -- Postselectorreferences replaced by LinkScoper and FrontierScheduler references -- before they can be used withHeritrix 1.6.0 (Referencing a non-existent Postselector in an order file usually shows as -50 fetch statusin crawl.log).

12.2.2. Web Console

The layout and terminology of the web Console and header have been changed, and new readouts added.Most notably, "Crawler Status" and "Job Status" information have been moved to separate boxes, withthe controls for each at the top of their respective boxes, near the current status information. Also, the"Crawling"/"Stopped" distinction in the crawler -- whether available pending jobs would be started aspossible -- has been renamed "Crawling Jobs"/"Holding Jobs" for clarity.




Add XMLExtractor(XML/RSS)



Add annotate whatrobots.txt wouldhave precluded


1069331[http://sourceforge.net/tracker/index.php?func=detail&aid=1069331&group_id=

Add hold pausedcrawl at 'end',allowing all in-progress ops



28





73833&atid=539102]


Add need way to de-lete overrides



Add Confusion:CrawlControllerand CrawlJobStates



Add alerts shouldshow currentprocessor



Add SURT needsfacelift



Add DecomposePostselector toScoping andSchedulingcomponents



Add [contribution]New fixed num-ber of queuespolicy


1173597[http://sourceforge.net/tracker/index.php?func=d

Add jmx api addi-tions



29









etail&aid=1173597&group_id=73833&atid=539102]


Add [contrib] Gener-alize/RefactorBDB Frontier

2005-04-05 stack-sf ck-heritrix


Add [contrib] UIstacktrace dump(Depends onJDK150)



Add Post 1.4 De-precate filterscope and re-move post 1.6.



Add Quick resumewithout real re-covery / Check-pointing

2005-04-27 karl-ia ck-heritrix


Add [contrib] Showestimated re-maining time



Add add 'exhausted'queue count tofrontier report


1204644[http://sourcefor

Add add 'memoryused' to pro-

2005-05-18 kristinn_sig gojomo


30









ge.net/tracker/index.php?func=detail&aid=1204644&group_id=73833&atid=539102]

gress-statist-ics.log


Add add Candid-ateURI para-meter to UriU-niqFil-ter.forget()



Add [contrib]ThreadLocal-ver-sion of Tex-tUtil.getMatcher



Add [contrib]WorkQueueFrontier: StoreallQueues inRAM if poss.



Add List based URI-RegExprFilter



Add [rfe-contrib]Add Stacktracedump to To-eThread.report()



Add CrawlURI seri-alizationbloated; shouldbe slimmed



31









Add Cookies arethread trafficjam andmemory hog



Add garbage hotspot: Serial-Binding &FastOutput-Stream.bump()



Add [contrib] Adddebugging aidfor BDBRuntimeExcep-tionWrapper



Add seed report ofredirect shouldshow where to



Add Rotation ofcrawl logs



Add BdbWorkQueueorigins shouldbe based on fullclassKey



Add Expose the bdbje 2.0 jmx inter-face



32









9102]


Add new alreadyIn-cluded option:Bloom filterbased



Add Add to queue-assignment-policy withoutcompile



Add More than oneHeritrix in-stance in a JVMinstance



Add MultimachineCrawl SplitterProcessor



Add add dummyitems at headsof BDB queuesfor performance



Add IP geolocationbased scoping



Add per-crawler'load' summarynumbers



33









208&group_id=73833&atid=539102]


Add Checkpointingfixes/improvements



Add [uuri] When'generous'mode, don't en-code curly-brackets



Fix Serious errorduring crawlingdid not producean alert

2004-06-03 gojomo kristinn_sig


Fix [UURI] escapedabsolute pathnot valid



Fix timing issue oncrawl-start &"run time" stat



Fix Other than de-fault seeds pathis not used



Fix hard-coding ofjob dir instate.job makes



34










moves awkward


Fix angle-bracketsin URIs thwartfrontier report



Fix [uuri] '$' in pathgets scheduled,spawnsqueueing error



Fix reducing max-toe-threads hasno effect



Fix Text incorrectin WUI whencreate new pro-file

2004-12-11 nobody zhousp


Fix max-trans-hops=0generates -63 incrawl.log



Fix RIS still open;ThreadLocal-Connection-Manager asyncclose


1113410 Fix NPE 2005-01-31 gojomo stack-sf


35










readalert_jsp._jspService


Fix ARCWriterlength is wrong(But coherentgzip record)



Fix frontier reportConcurrent-Modifica-tionException

2005-02-09 gojomo frodobay


Fix Localize Stack-OverflowErrorin Extractors



Fix Aggressive ex-traction of `for'attributes



Fix NPE inFetchDNS,caused byUURI

2005-04-22 nobody ck-heritrix


Fix OOME guardagainst pages ofthousands oflinks



36









Fix WaitEvaluatorsaccidentally re-moved fromPro-cessors.options



Fix minor CSStypo, wrongtext font.



Fix Build w/ 150jdk won't rununder 14x.



Fix Web UI recov-ery manglespaths

2005-05-12 karl-ia karl-ia


Fix can't changecost policy mid-crawl



Fix BdbFrontier,serious excep-tion - Latch-NotHeldExcep-tion

2005-05-17 karl-ia kristinn_sig


Fix JMX remotecommand 'stop'leaves zombiecrawler

2005-05-17 stack-sf karl-ia


37









9099]


Fix uri-errors.log:old timestamps,too many errors



Fix 'custom' robotspolicy doesn'twork



Fix NPE whenviewing crawlreport

2005-05-19 karl-ia kristinn_sig


Fix [arcwriter] Re-cord w/ emptybody on OOME



Fix seeds listedwithout scheme,but with path,being ignored



Fix CachedBdbMapNPE killing offthreads.



Fix Failed URIsshould be 'free'(no cost againstqueue budget)

2005-05-26 gojomo kristinn_sig


38









046&group_id=73833&atid=539099]


Fix Re-playCharSequenceFactory: Un-expected re-sponse bodyoffset

2005-05-27 karl-ia ck-heritrix


Fix URIExceptionin deserializa-tion, postCrawlURI slim-ming



Fix UURI handlingof inconsistentescaping makesbroken instance



Fix ThreadLocalHt-tpConnection-Manager startsa non-daemonThread



Fix recovery fillsheritrix_outwith "RelativeURI but nobase..



Fix Non-canonicalseed URLs needbetter reporting

2005-06-08 karl-ia karl-ia


Fix GzippedInput-Stream class isnot thread-safe



39











Fix CookieSpec in-terface modific-ation breaks Ig-noreCook-iesSpec



Fix ARCReader:Bad URI escap-ing for tab char-acter



Fix odd preencodedURIs generateerror on deseri-alization



Fix broad crawlsslow; mostthreads stuckretrying missingsites



Fix 'treat seed redir-ects as newseeds' not work-ing



Fix no rule for de-cidingscope toalways crawlseeds


1219715 Fix [patch] Signa- 2005-06-13 nobody ck-heritrix


40










ture changebroke Buck-etQueueAssign-mentPolicy


Fix NPE je-2.0entryToOb-ject(SerialBinding.java:82)



Fix ExtractorHTMLexcessive tempstrings / OOM



Fix reports (web uiand to disk)don't scale



Fix unicode/idn do-main names fail(seeds andmore?)- pun-ycode



Fix strip-www can-on. rule causesfailed crawl ofnetarkivet.dk



Fix '@' in URI pathconfuses SURT(bad queues,scope probs,etc)



41









Fix DomainSensit-iveFrontierbroken by uri-included-struc-ture



Fix BdbUriUniqFil-ter URI finger-print somewhatcollision prone



Fix CandidateURIserialization 'de-codes' UURI



Fix nonstandardport URIs failwith '-50': Surtqueue policybug



Fix DNS prereqproblems (-50fails/repeats):calculateInser-tKey



Fix IdentityCach-ingMapTestFAILED: cachenot cleared ap-propriate



Fix IdentityCach-ingMapTestfails on fedora



42









9099]


Fix WorkQueueFronti-er.kickUpdateClassCastEx-ception / unre-tiring



Fix Possible dead-lock situationwith ARFronti-er



Fix Cannot setcachePercent-age in bdbjeJMX bean



Fix Link ClassCas-tException



Fix [arcreader]Fails on recordsthat only haveheaders



Fix Connection re-set error withWebSTAR/3.0.2 web serve



Fix over-escaping(of '%', etc)compared tobrowsers



43









747&group_id=73833&atid=539099]


Fix custom ro-bots.txt NPE in1.4.0

2005-07-31 gojomo efc


Fix -5000 out-of-scope pre-conditions; -50failure



Fix host with trail-ing '.' in seedsruins impliedSURT



Fix Deferred URIshould be 'free'(no cost againstqueue budget)



Fix create job:empty seedsbox, broken set-tings tab



Fix already-seentest not working(1.4.0); canon-icalization



Fix Prune classesdeprecated in1.4.0



44











Fix NotificationAFTER reportshave been cre-ated



Fix mimetype-re-ports squashestogether url-count and byte-count



Fix Help links touser and devmanuals 404



Fix StatisticsTrack-er writes 'notset'. Crawl.logsays 'no-type'



Fix mime type re-port not consist-ent withcrawl.log

2005-09-14 ia_igor ia_igor


Fix ExtractorUni-versal skipsURI with '_'


1323281 Fix StringIndex- 2005-10-10 gojomo gojomo


45










OutOfBoundsin SurtAuthor-ityQueueAs-signmentPolicy


Fix BDB ArrayIn-dexOutOf-BoundsExcep-tion; corruptqueue



Fix IDNAExcep-tion: String toolong / fixupDo-mainLabel instdout



Fix NPE in Crawl-JobHand-ler.startNextJobInternal



Fix implied-HTTPseeds with portnumbers get -5000 errs



Fix 'expert' viewtoggle does in-advertent sub-mit



Fix time-based av-erages wrongafter checkpoint



46









Fix delete functionin "View orEdit FrontierURIs" Unsup-ported



Fix Laxuri code en-codes tildes.



Fix Investigatespate of recentOOMEs



Fix Double JMX re-gistration prob-lem on remotecreation



Fix DevUtils.extraInfo shows 2 le-gend lines, nocontent



Fix transient JSPNPE after start-ing job



Fix NPE in quotaE-nforcer, if host-name for URIcan't be re-solved

2005-11-29 nobody svc


47









9099]


Fix NPE in QuotaE-nforcer



Fix properties-based suppor-ted/ignoredscheme settingbroken



Fix ArithmeticEx-ception: / by 0WorkQueueFronti-er.averageDepth


1325230[http://sourceforge.net/tracker/index.php?func=detail-amp;aid=1325230&group_id=73833&atid=539099]

Fix jmx import ofseeds not work-ing

2005-10-1214:29

karl-ia stack-sf


Fix Queue countswrong aftercheckpointing

2005-10-1209:10

karl-ia stack-sf


Fix Out Of Memoryafter creatingmutliple jobs

2005-02-1507:51

karl-ia nobody


Fix "Failed getPath"alerts (from Ro-

2005-10-1002:42

karl-ia gojomo


48




http://sourceforge.net/tracker/index.php?func=detailamp;aid=1325230&group_id=73833&atid=539099





ge.net/tracker/index.php?func=detail-amp;aid=1322280&group_id=73833&atid=539099]

botsExclusion-Policy)


Fix NumberForma-tException inFetchHT-TP.innerProcess

2005-10-1002:34

karl-ia gojomo

13. Release 1.4.0 - 2005-04-28Much improved memory usage, new scoping/filter model, and a new revisiting frontier. Over 90 bugsfixed.


13.1.1. Glibc 2.3.2 and NPTL

NPTL is the 'new' linux threading model. It replaces linuxthreads the 'old' model. You can tell you'rerunning NPTL if your java process shows as one process only in the process listing. Wwith linuxthreads,all java threads show as distinct linux processes. Linux threading is integral to glibc.

On rare occasions we've seen the crawler hang without obvious explaination when running with NPTLthreading on linux. Doing a thread dump on the hung crawler, one version of the hung crawler hasthreads waiting to obtain a lock that no one apparently holds. Our reading has these rare, crawl-killing,hangs as a problem in glbc2.3.2 when running with NPTL (NPTL 0.60) (We used to hang frequently butworkarounds seem to have mitigated the frequency of lockup making it extremely rare). An upgrade toglibc2.3.3+ seems to do away with these hangs. Glibc2.3.3 has NPTL 0.61. Fedora3 has glibc2.3.4. If anupgrade is not possible -- for example, the new glibc is not currently available for debian -- you can dis-able NPTL and run with old threads by setting the environment variableLD_ASSUME_KERNEL=2.4.1 (You can set this environment variable on a per process basis).

NPTL is usually the default threading model on linux and is usually what you want -- threads are morelightweight and java throughput seems to be slightly higher with NPTL enabled. Various are the ways inwhich you can see which threading model you are using. Do an ldd on the java executable to see whatshared libraries its using. Note the location of the glibc shared library. ExecutingPATH_TO_GLIBC/lib.so.6, usually /lib/lib.so.6, will list details on glibc. Look in the list-ing for either 'nptl' or 'linuxthreads'. On debian systems, lib.so.6 is not executable but you can make itso. You can also do the following to determine library versions and which threading you are using: %getconf GNU_LIBC_VERSION and % getconf GNU_LIBPTHREAD_VERSION.

See [ 1086554 ] glibc 2.3.2 NPTL hang (Was bdbfrontier stall in...)[http://sourceforge.net/tracker/index.php?func=detail&aid=1086554&group_id=73833&atid=539099]for more on the issue.


49



13.1.2. [1093962] SSL handshake fails when server requests switchto SSL V3[http://sourceforge.net/tracker/index.php?func=detail&aid=1093962&group_id=73833&atid=539099]

When connecting to a secure server, if the server wants to switch from SSL V2 to SSL V3 when client isusing a SUN JVM, the connection fails. See issue 1093962for more.

13.1.3. Using old jobs or profiles with 1.4

You'll need to make one change to make your old order.xml files and profiles to run with Heritrix 1.4.x.Below is a diff that shows the change that needs to be made (The type of the path changed fromstring to stringList):

+++ order.xml 2005-02-01 13:12:34.000000000 -0800@@ -162,7 +162,9 @@

<string name="prefix">BT</string><string name="suffix"></string><integer name="max-size-bytes">100000000</integer>

- <string name="path">arcs</string>+ <stringList name="path">+ <string>arcs</string>+ </stringList>

<integer name="pool-max-active">5</integer><integer name="pool-max-wait">300000</integer>

</newObject>

13.1.4. [ 1119644 ] frontier ConcurrentModificationException[https://sourceforge.net/tracker/?func=detail&atid=539099&aid=1119644&group_id=73833]

Sometimes you'll get a ConcurrentModificationException exception when you go to view or refresh theFrontier's report page. Workaround is to retry. The page should eventually come up.

13.1.5. New ARC file suffix

Pre-release 1.2.0, currently open ARC files that are being written to by the crawler were differentiatedby an '.open' suffix. When the crawler finished writing, the suffix was removed. A new suffix has beenintroduced -- '.invalid' -- which the crawler will use to mark ARC files it thinks suspect -- usually be-cause there was an IOException thrown during the writing of an ARC Record. Such ARCs need to bechecked for validity. Run % gzip -t and % ARCReader --strict against all files with an'.invalid' suffix -- and any unclosed '.open' files present after a crawl has ended -- to check for corrup-tion.

13.1.6. DNS lookups fail (-6 in crawl.log)

[1149470] all DNS attempts fail -6[https://sourceforge.net/tracker/index.php?func=detail&aid=1149470&group_id=73833&atid=539099]discusses badly-formatted DNS records returned on windows platform that Heritrix fails to parse and itincludes a pointer to a mailing list discussion of failed lookups on non-english windows. The issue in-cludes description of a workaround.

13.1.7. FatalConfigurationException creating new job based on old


50



https://sourceforge.net/tracker/?func=detail&atid=539099&aid=1119644&group_id=73833


Older SUN JVMS -- pre-beta3 versions of the SUN JVM 1.5.0 for instance -- had an issue using niocopying files. Try upgrading your JVM. See [1178102] FCE on creation of new job based on job w/overrides[http://sourceforge.net/tracker/index.php?func=detail&aid=1178102&group_id=73833&atid=539099]for more on this.

13.1.8. OutOfMemoryErrors (OOMEs)

Unusual pages -- pages of unorthodox structure, pages that contain thousands upon thousands of links --will on occasion produce OOMEs.

There have been improvements regards memory usage running multiple jobs in series, Section 14.1.3,“Running more than one job in series throws OOME”, but starting up a new job after a long-running jobcan prompt OOMEs. Workaround for now is to restart Heritrix between the running of big jobs.

13.2. Changes

13.2.1. Berkeley DB Based Frontier

The BdbFrontier -- a frontier that keeps its queues of URIs in Berkeley DB Java Edition[http://www.sleepycat.com/products/je.shtml] databases -- has been made the default Frontier. Othercore datastructures such as the queue of 'alreadyseen' URIs have also been moved into bdbje databases.

13.2.2. The IP in dns ARC Records

Dns entries in ARCs look like this:

dns:www.archive.org 207.241.238.254 20050310233154 text/dns 58 20050310233154www.archive.org. 1600 IN A 207.241.224.241

The above record is for the lookup of www.archive.org.

Previous to 1.4.0, the IP used on the ARC Record metaline -- the first line of an ARC Record entry(207.241.238.254 in the above example) -- was the IP of the host looked up. As of 1.4.0, we write the IPof the dns server that returned us the address looked up. Previous to this there was no recording of thednsserver IP.

13.2.3. AdaptiveRevisitFrontier

A new, experimental Frontier with configurable revisiting policy and tools for noticing page change, etc.

13.2.4. DecidingScope and DecidingFilterA.K.A New Scoping Model

A new, experimental scope and filter that allow the user to pick and choose from an assortment of ready-made decision rules and have each rule applied in an orderable sequence. The last non-PASS decisionstands as the aggregate decision for the decide rule sequence.

13.2.5. Crawl Size Upper Bounds Update

Memory usage has been improved in this release. Previously RAM-based datastructures that grewwithout bound now are disk-backed kept in berkeley db databases. Where previous, see Section 17.1.1,“Crawl Size Upper Bounds”, Heritrix was unsuited for broad crawling, while still experimental, usingdefault memory settings -- a heap of 256m -- broad-crawls of 5 to 6 days before encounteringOutOfMemoryErrors (OOMEs) are now possible; longer if more heap is assigned. Where 10k hosts wasan upper bound on narrow domain- or host-scoped crawls, now, using the default heap size, it should


51



http://www.sleepycat.com/products/je.shtml

now be possible to do 500k+ hosts.

Long-running crawls that encounter hundreds-of-thousands of hosts over the life of a crawl, or crawlsstarted with hundreds-of-thousands of seeds, continue to throw OutOfMemoryErrors because there arestill a few RAM-based datastructures that grow without bound left in Heritrix; the lists of queue namesand internal structures inside 3rd party libraries used by Heritrix. These last few items we intend to ad-dress in a later release.

13.2.6. IBM JVM Redux

Testing with IBM JVM 1.4.2 (Classic VM (build 1.4.2, J2RE 1.4.2 IBM buildcxia32142sr1a-20050209 (JIT enabled: jitc))) using Heritrix 1.4.0, the SSL problemdescribed in Section 14.1.1, “IBM JVM” is no longer present (All of our crawling of the last couple ofmonths has been done on the latest SUN 1.5.0 JVMs).

Table 6. Changes


958061[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539102&aid=958061]

Add [Post 1.0] Newscoping model



Add Add links to is-sue tracking/RFE to Heritrix'webapp



Add Integrate revis-iting frontier

2005-02-09 kristinn_sig stack-sf


Add One-click re-cover


1078008[http://sourceforge.net/tracker/index.php?func=d

Add Enable crawl-end at targetcompressed-ARC-data size



52

http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539102&aid=958061






etail&group_id=73833&atid=539102&aid=1078008]


Add Need 'deleteprofile' option(like delete job)



Add A 'dat' maker; Ascript to dumplinks



Add Add refererheader



Add [contribution]SingleConnec-tionManager,range and closehdrs



Add Dates in logsare unreadable.



Add Extractorsshould not ex-tract if linksalready extrac-ted



Add Pause and alerton low-disk



53









ge.net/tracker/index.php?func=detail&group_id=73833&atid=539102&aid=1047437]

conditions


Add Add info tocandidateURIbefore schedul-ing



Add Change arcdownload dirmid-crawl



Add Stopping, paus-ing, checkpoint-ing from com-mand line/scripts



Add [jmx] clientpword and al-ways start jmxserver



Add Move BDB tocore of Heritrix



Add [ARCReader] Ifgarbage on endof record, reportand skip it



54









Add 'Economic'frontier whichdefers low-value URIs



Add Evaluate Berke-ley DB Frontier



Add Update com-mons-pool,commons-col-lections, itextjars



Add ARC writerpool config. towrite multipledisks



Add Command-lineinsertion ofURLs



Add Make auto seedadd on redirectoptional (if hap-pens at all)


1002707[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539102&aid=100

Add Fix heritrixshutdown(From Luca)



55









2707]


Add Recoveryshould option-ally retain fail-ures ('Ff')



Add HTTPRecord-er's default buf-fer sizes shouldbe configurable



Add Untangle herit-rix from jetty



Fix NPE in Ar-cWriterPro-cessor.writeDns()



Fix 'submodules'map-edits notworking foroverrides/re-finements



Fix NPE in Fast-BufferedOut-putStream.close


1184102[http://sourceforge.net/tracker/index.php?func=detail&group_id

Fix Frontier queuestotal still goesminus



56









=73833&atid=539099&aid=1184102]


Fix ARCWriterAsynchron-ousCloseExcep-tion



Fix CME addingfilters whilecrawling



Fix job config: set-tings 're-move'-component-then-submitlost job



Fix hosts-report.txtis empty

2005-04-04 stack-sf danavery


Fix Delete URIsfrom frontierbroken(CachedBdbBigMap.values()?)



Fix FCE on creationof new jobbased on job w/overrides



Fix hung bdb(12115 redux)



57









dex.php?func=detail&group_id=73833&atid=539099&aid=1178103]


Fix CachedBdb-BigMapdouble-close infinialize()



Fix RIS#readFullyOrUntil IOE/timeout



Fix all DNS at-tempts fail -6

2005-02-22 nobody jsleeman


Fix Flash SWF Ex-tractor Unex-pected end ofinput



Fix npe in extract-orjs doing broadcrawl w/ HEAD



Fix Heritrix 1.3.0crashes hard(JVM SIGSEV)


1103015 Fix If filter in main 2005-01-15 nobody frodobay


58









[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=1103015]

scope disabledheritrix abortsimme


Fix Links not ex-tracted frommislabelled(text/plain)MIME type



Fix Lost crawl jobafter terminaterunning jobwith jobspending



Fix www-strip ca-nonicalizationunintended ex-clusion of redir-ect



Fix DNS records inARCs shoulduse DNS serverIP



Fix Crawler notmaking pro-gress -- threaddeadlock

2005-03-05 stack-sf ia_igor


Fix isMultibyteEn-coding: Un-caught Unsup-portedOpera-tionException



59









Fix MultiThreaded-Connection-Manager bottle-neck



Fix missing spacein progress-stat-istics.log



Fix npe in Extract-or-HTML#innerProcess



Fix "Illegal re-sponse bodyoffset" in Re-playCharSequenceFactory



Fix ensure IPsmatch fromDNS, used inHTTP, loggedin ARC



Fix swf extractorflash lib printsglyphcount onstdout



Fix crawl.logtimestamps out-of-order



60









7924]


Fix sometimes jobbased-on otherjob uses olderjob name



Fix seeds text areatruncates seeds;big seed listsbreak config



Fix OOM hit veryearly broad-crawling



Fix UI list-orderinginconsistent



Fix UI/Settings -Expert Toggleloses user data



Fix OOM in postse-lector

2005-02-26 nobody orion2598


Fix ARCWritergzip deflatehang



61









=73833&atid=539099&aid=1068403]


Fix ARCWriteralerts if Con-tent-Type isnull



Fix Bad synchron-ization causesNPE in Statist-icsTracker



Fix ARCWriter'Gap' errorsshould be moreprominent



Fix Change in Ex-tractorHTMLtriggers Null-PointerExcep-tions



Fix StackOver-flowErrorshouldn't killcrawl



Fix [Flash] OOMEson a particularURL



Fix unwritableARCs directorybarely notice-



62










able


Fix "&amp" con-verted to "&" inpreselectoroverride regex

2004-09-07 gojomo danavery:


Fix remove profilefunction inWUI?

2004-12-11 nobody zhousp


Fix deleting all(?)from queue cor-rupts frontier,kills crawl



Fix ExtractorCSSregexp taking'forever' onsmall document



Fix FetchDNSdoesn't work(bug in dnsjava)



Fix Redirect prob-lem (Stopscrawling after3)


1119686 Fix oversight in 2005-02-09 nobody frodobay


63










CrawlURI;missing checkfor null


Fix [uuri] portStringIndex-OutOfBound-sExceptionn



Fix NPE inROS#record



Fix Old profile/jobswon't work withHEAD (1.4)



Fix First arc recordlength is off byone



Fix PDFParserURL extractionbug

2005-02-07 nobody benlitchfield


Fix User Agent istolowercased



64









Fix Exception inModules Tab



Fix Hung Thread inStatisticsTrack-er



Fix Failed createnew job basedon job with ab-solute settings

2005-01-22 nobody frodobay


Fix Long randompauses whereno progress ismade



Fix InvalidJob-FileException:Status .. 'RUN-NING'



Fix heritrix wontstart with fedoracore 3



Fix crawl.loghashes wrongfor captures >64K



65









2135]


Fix deadlock in ip-politeness re-queueing



Fix SURTs-from-seeds maylack trailingcomma



Fix JS extr. doesnot parse spec.links starting w/./ or ../

2005-01-12 nobody ia_igor


Fix update to [1100467 ]maven 1.0.2build problem



Fix Update ant andhttpclient jars



Fix Re-playCharSequence.toString() isbroken



Fix [robots] ro-bots.txt mid-fetch abortedgives open ac-cess



66









=73833&atid=539099&aid=1093627]


Fix midfetch abortdoesn't



Fix [uuri] String in-dex out ofrange: 0



Fix glibc 2.3.2NPTL hang(Was bdbfronti-er stall in...)



Fix [uuri] Under-score in hostmesses up portparsing



Fix better/longerdns retries onlookup failure



Fix NPE in Server-Cache



Fix reducing max-toe-threads hasno effect



67











Fix NPE in Tex-tUtils.freeMatcher()



Fix heritrix.log ig-nored



Fix Edit configura-tion in UI givesNPE



Fix terminatedcrawl still hog-ging memory,causing OOM



Fix quick-overrideaccepts domainw/spaces, lostcheckboxes



Fix Browser hangswhen hundredsof seeds


1047396 Fix OOM in Bdb- 2004-10-14 nobody gojomo


68










Frontier/nio.Bits -- withplenty of heapleft


Fix DomainSensit-iveFrontier nev-er finishes



Fix Upgrade bdbje1.7.0 (WAS:Checkpointerthread ...)



Fix bdbfrontier Nolocks available



Fix Deletedpending jobsshow aspending in re-start.


14. Release 1.2.0 - 2004-11-16Added IP-based politeness, configurable URI-canonicalization, and mid-fetch abort. Lots of Bug fixes.

14.1. Known Limitations

14.1.1. IBM JVM

The IBM JVM generally is more performant than SUN JVMs. It also emits more detailed heap dumps.That said, new Heritrix 1.2.0 features may not work on the IBM JVM.

14.1.1.1. HTTPS


69





Heritrix 1.2.0 uses the new HttpClient 3.0x library which allows the setting of socket read timeouts.Connections to https sites fail if using the IBM JVM.

The IBM JVM 141 (cxia321411-20030930) NPEs setting the NoTcpDelay.

java.lang.NullPointerExceptionat com.ibm.jsse.bf.setTcpNoDelay(Unknown Source)at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:683)at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1328)

Using the IBM JVM 142, its saying SSL connection not open when we go to use inputstreams:

java.net.SocketException: Socket is not connectedat java.net.Socket.getInputStream(Socket.java:726) at com.ibm.jsse.bs.getInputStream(Unknown Source)at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:715)at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1328)

Newer versions of the httpclient library may address this (Current version is alpha2).

14.1.2. Jobs don't show in UI when a bunch are run back-to-back

If more than one job waiting in the queue of pending jobs, then the second job often won't show in theUI; The UI says its running but its not possible to see a status bar on the running job. See [ 1024120 ]Lost crawl job after terminate running job with jobs pending[https://sourceforge.net/tracker/?func=detail&aid=1024120&group_id=73833&atid=539099]. For now,the workaround is to study the running job by viewing the crawl job logs on disk (Oddly, the 3rd queuedup job will start to show in the UI again).

14.1.3. Running more than one job in series throws OOME

OutOfMemoryExceptions are frequent when jobs are run in series. [ 1055592 ] terminated crawl stillhogging memory, causing OOM[https://sourceforge.net/tracker/index.php?func=detail&aid=1055592&group_id=73833&atid=539099].For now, restart Heritrix between the running of jobs.

14.2. Changes

Table 7. Changes

ID Type Summary Open Date By

1067095[http://sourceforge.net/track-er/in-dex.php?func=detail&group_id=73833&atid=539102&aid=1067095]

Add Hang in http fetch-er when mid-fetchaborts

2004-11-15 stack-sf

1066804[http://sourceforge.net/track-

Add Allow specificationof heritrix_out.logfilename

2004-11-15 stack-sf


70

https://sourceforge.net/tracker/?func=detail&aid=1024120&group_id=73833&atid=539099







er/in-dex.php?func=detail&group_id=73833&atid=539102&aid=1066804]


Add IP-based politeness 2004-10-28 gojomo


Add Recover fromcrawl initializedwith a recovery log

2004-10-26 stack-sf


Add Import gzipped ornon-gzipped recov-ery log

2004-10-26 stack-sf


Add Add bdbalreadyseen optionto hostsqueuesfron-tier

2004-10-19 stack-sf


Add Force generation ofreport files

2004-06-16 stack-sf


Add Scripts to generateend-of-job reports

2004-08-17 danavery


71









Add [Need feedback]"Done with ARCfile" event

2004-07-09 stack-sf


Add Logging of scope-rejected URIs

2004-10-11 stack-sf


Add HTTPClient shoulduse supplied IP /avoid DNS lookup

2004-02-23 stack-sf


Add Setting of In-teger.MAX_VALUE is ugly

2004-02-23 stack-sf


Add canonicalization ofURIs for alreadyIn-cluded testing

2004-02-18 stack-sf


Add Allow operator-configured mid-HTTP-fetch filters

2004-04-23 stack-sf

1037891[http://sourceforge.net/track-er/in-dex.php?func=detail&group_id=73833

Add Cmdline defaults inproperties file

2004-09-30 stack-sf


72









&atid=539102&aid=1037891]


Add Upgrade httpclientto 3.0.x

2004-09-29 stack-sf


Add Update build to usemaven 1.0

2004-07-19 stack-sf


Add Figure what pro-filer to use

2004-08-02 stack-sf


Fix http and https pre-requisites conten-tion

2004-11-11 stack-sf


Fix Seed to SURT cov-ersion issuesI

22004-11-08 stack-sf


Fix ServerCacheHashMaps accessthread-safetyI

22004-11-06 gojomo


Fix Can't open logs ofold jobs post-restartin UII

22004-11-04 stack-sf


73











Fix Non-default 'logs'location doesn'tshow in web UI

2004-11-010 stack-sf


Fix IMG 'lowsrc' maynot be extracted

2004-11-010 stack-sf


Fix completed crawlsshow as 'aborted byuser'

2004-10-270 gojomo


Fix MultiThreadedHt-tpConnectionMan-ager https alreadyconnected

2004-11-020 stack-sf


Fix recovery log of re-covered crawl in-sufficient to recov-er

2004-10-220 stack-sf


Fix Some dates areGMT, others arenot *

2004-03-020 gojomo


74









Fix FlushingCrawlServers prob-lematic *

2004-05-210 gojomo


Fix Threads contendfor scratch files(after kill/readFully/Gap)

2004-10-220 gojomo


Fix incorrect number oftotal active threads*

2004-09-230 gojomo


Fix diskincludedfronti-er performance isawful

2004-07-30 gojomo


Fix better/longer dnsretries on lookupfailure

2004-10-08 gojomo


Fix ExtractorHTMLtakes forever onworst-case HTML

2004-10-20 gojomo


Fix ExtractorJS takesforever on worst-case JS

2004-10-21 gojomo


75









&atid=539099&aid=1051916]


Fix jdk required (docimplies jre)

2004-10-19 stack-sf


Fix prerequisite hyster-esis/robots ahead ofdns

2004-09-30 gojomo


Fix Crawl upper time/size bounds ig-nored

2004-08-24 gojomo


Fix timing issue oncrawl-start & run-time stat

2004-08-02 gojomo


Fix inactiveQueues-MemoryLoadTar-get mechanism be-haves poorly

2004-08-02 gojomo


Fix DNS URIs don'tget override set-tings

2004-10-11 gojomo


Fix Gzipped recoverlog corrupt at end;last < 32K unre-

2004-07-26 gojomo


76










coverable


Fix No crawl if host-queues-memory-capacity =0

2004-07-26 stack-sf


Fix frontier report un-usable in bigcrawls; frontierinfo needed

2004-08-02 gojomo


Fix Build fails: "rws"mode and Mac OSX interact badly

2004-07-02 stack-sf


Fix fatal runtimeexcep-tions in frontiergive no info in webUI

2004-07-30 gojomo


Fix seed parser *too*lenient

2004-06-01 johnerik


Fix Auth unsupportedlogged to console

2004-06-25 stack-sf


77









Fix bad queue keys:shouldn't be URIs;should be handledbetter

2004-08-02 stack-sf


Fix UURIFact-ory.validateEscaping() -> IllegalArgu-mentException

2004-10-13 stack-sf


Fix ARCReadercrashes if zero-length gzip record

2004-10-12 stack-sf


Fix [UURI] Catch bad-encoding earlier

2004-08-02 stack-sf


Fix PathDepthFilter in-nerAcceptsSEVERE log:"Failed getPath..."

2004-09-28 stack-sf


Fix Unnecessary to-String() in Extract-or-HTML.processScriptCode()

2004-10-12 gojomo


Fix Domain names in'overrides' are notin alphabetical or-der

2004-10-11 kristinn_sig


78









&atid=539099&aid=1044527]


Fix If CC timesoutselftest, no buildfailed message

2004-08-19 stack-sf


Fix selftest hanging be-cause no crawl stopevent

2004-08-19 stack-sf


Fix CrawlStateUpdater- NullPointerEx-ception

2004-04-08 stack-sf


Fix NoSuchElemen-tException in URIqueues halts crawl-ing

2004-06-15 gojomo


Fix [UURI] >2047AFTER escaping(Stops crawl)

2004-09-23 stack-sf


Fix crawl.log has URIswith spaces in them

2004-08-17 stack-sf


Fix unfetchable URIschemes shouldnever be queued

2004-05-31 gojomo


79











Fix KeyedQueue serv-er<->key mismatchnoted: pf-buser<->mprsrv.agr

2004-09-20 stack-sf


Fix NPE reading over-ride

2004-09-20 stack-sf


Fix Wrong handling ofdate in ARCRe-cordMetaData

2004-09-20 johnerik

15. Release 1.0.4 - 2004-09-22Bug fix.

15.1. Changes

Table 8. Changes

ID Type Summary Open Date

1010966[http://sourceforge.net/tracker/in-dex.php?func=detail&group_id=73833&atid=539099&aid=1010966]

Fix crawl.log has URIs withspaces in them

2004-08-17

16. Release 1.0.2 - 2004-09-14


80





Bug fixes.

16.1. Changes

Table 9. Changes

ID Type Summary


Fix old crawls stick around, consum-ing memory


Fix Terminating paused crawl leaveszombie threads


Fix Excessive ARCWriterPooltimeouts


Fix Per host overrides not taking ef-fect.


Fix document size limit not working


Fix UURI.length() > 2k


Fix crawl.log has URIs with spacesin them stack-sf

17. Release 1.0.0 - 2004-08-06Added new prefix ('SURT') scope and filter, compression of recovery log, mass adding of URIs to run-ning crawler, crawling via a http proxy, adding of headers to request, improved out-of-the-box defaults,hash of content to crawl log and to arcreader output, and many bug fixes.


81








17.1. Known Limitations

17.1.1. Crawl Size Upper Bounds

Heritrix 1.0.0 uses disk-based queues to hold any number of pending URIs bounded only by availabledisk space, but still relies on in-memory structures to efficiently track all discovered hosts and previ-ously-scheduled URIs. Crawls whose total scheduled URIs or discovered hosts exhaust all availablememory will trigger out-of-memory errors, which freeze a crawl at the point of the error.

With the default settings, and an assignment of a 256MB Java heap to the Heritrix process, crawlingwhich discovers up to 10 000 hosts, and schedules over 6 000 000 URIs, should be possible. Discoveryof higher numbers of URIs/hosts will likely trigger out-of-memory problems unless a larger java heapwas assigned at startup.

Broad crawls -- those using the BroadScope or ranging over domains with many subdomains -- can eas-ily and quickly exceed these parameters. Thus broad crawls in Heritrix 1.0.0 are not recommended, ex-cept for experimental purposes.

Narrower crawls, restricted to specific hosts or domains a limited number of subdomains, can run for aweek or more, collecting millions of resources. Larger heaps can allow crawls to run into the tens of mil-lions of collected URIS, and tens of thousands of discovered hosts.

An experimental alternate Frontier, the DiskIncludedFrontier, is also available via the 'Modules' crawlconfiguration tab. It uses a capped amount of memory plus disk storage to remember any number ofscheduled URIs, but its performance is poor and it has not received the same testing as our default Fron-tier. The memory cost of additional discovered hosts continues to rise without limit when using aDiskIncludedFrontier.

Future versions of Heritrix will include other frontier implementations allowing larger and unboundedcrawls with minimal performance penalties.

17.1.2. [ 958055 ] Seed ConcurrentModificationException[https://sourceforge.net/tracker/?func=detail&aid=958055&group_id=73833&atid=539099]

Its possible to get ConcurrentModificationsException editing options on a running crawl.

17.1.2.1. Workaround

Pause the crawl when making changes to crawl options.

17.1.3. [ 984390 ] Build fails: "rws" mode and Mac OS X interactbadly[https://sourceforge.net/tracker/?func=detail&aid=984390&group_id=73833&atid=539099]

On macintoshes and linux kernel version 2.6, heritrix fails to build (unit tests fail).


See issue, [ 984390 ] Build fails: "rws" mode and Mac OS X interact badly[https://sourceforge.net/tracker/?func=detail&aid=984390&group_id=73833&atid=539099], for sourcecode workaround edit.


82





17.1.4. [ 955975 ] Build fails: JVM and kernel 2.6+ (Was 2 tests fail...)[https://sourceforge.net/tracker/index.php?func=detail&aid=955975&group_id=73833&atid=539099]

Heritrix fails to build on linux kernel 2.6.


Build fails unless you use a JDK in advance of pedigree 1.5 beta 2 (It works with jdk1.5.0-rc). See [955975 ] Build fails: JVM and kernel 2.6+ (Was 2 tests fail...)[https://sourceforge.net/tracker/index.php?func=detail&aid=955975&group_id=73833&atid=539099]and above [#984390].

17.2. Changes

Table 10. Changes

ID Type Summary


Add Mass-add URIs to running crawland force reconsideration


Add SurtPrefix scope (and filter)


Add Specification of defaultCharSequence charset


Add crawl.log entries all on one line


Add Hash content-bodies, show inlogs (and future ARCs)


Add option to preference (quick-get)embeds

964493[http://sourceforge.net/tracker/index.php?func=detail&group_id=7

Add Compress recover.log


83




#984390








ID Type Summary

3833&atid=539102&aid=964493]


Add [UURI] 'http:///...' converted to'http://...'


Add enable use through HTTP proxy


Add Allow adding (subtracting?) httpheaders


Add Improved out-of-the-box defaults


Add ARCWriter makes FAT gzipheader


Add exponential backoff URI/host re-tries

- Fix Total data "written" isn't neces-sarily written (wording)

- Fix embeds within scope problem

- Fix NPE clearing alerts

- Fix arcmetadata repeated once forevery domain config

- Fix CCE deserializing diskqueue[Was: IllegalArgumentExcepti...]

- Fix no docs for recovery-journal fea-ture

- Fix Pause/Terminate ignored on 2.6kernel 1.5 JVM

- Fix Investigate "Relative URI but nobase"

- Fix User-Agent should be able tomimic Mozilla (as does Google)

- Fix referral URL should be stored inrecover.log


84







ID Type Summary

- Fix ToeThreads hung in FetchDNSafter Pause

- Fix robots.txt lookup for differentports on same host

- Fix Empty log percentages displayedas NaN%

- Fix UURI doubly-encodes %XX se-quences

- Fix Single settings change causes twoversions to be created

- Fix New IA debian image is 2.6(Was: Build fails: JVM and ...)

- Fix NPE in PathDepthFilter

- Fix [investigate & rule out] Threadreport deadlock risks

- Fix jetty susceptible to DoS attack

- Fix 'ignore' robots does not ignoremeta nofollow

- Fix URI Syntax Errors stop pageparsing.

- Fix NPE in ExtractorHTML/Tex-tUtils.getMatcher()

- Fix ARCReader: Failed to find GZIPMAGIC

- Fix javascript embedded URLs

- Fix NoClassDefFoundError whenstarting a job

- Fix Max number of deferrals hard-coded to 10.

- Fix Frontier report thread safetyproblems?

- Fix ARCReader hanging

- Fix log-browsing by regexpoutofmemoryerror

- Fix Deferred URLs due the DNSproblem -- Heritrix(-50)-Deferred

- Fix Assertion failures shouldn't bemore fatal than Runtime Exc.

- Fix min-interval is superfluous; re-move

- Fix crawl doesn't end when usingvalence > 1

- Fix Giant (in # of files) state direct-ory problematic

- Fix robots-expiration units, defaultwrong


85

ID Type Summary

- Fix NoSuchElementException inURI queues halts crawling

- Fix #anchor links not trimmed, andthus recrawled

- Fix arc's filedesc file name includes.gz

- Fix [denmark-workshop] Cookiemangling

- Fix HttpException: Unable to parseheader

- Fix bogus ARC-header when noContent-type

- Fix paths when crawling without UI

- Fix domain scope leakage

18. Release 0.10.0 - 2004-06-04Release for second heritrix workshop, Copenhagen 06/2004 (1.0.0 first release candidate). Added site-first prioritization, fixed link extraction of multibyte URIs, added metadata to arcs as xml, changed arcnaming template, new user and developer manuals, added basic/digest auth and http post/get login facil-ity, and added help to UI. Bug fixes.

18.1. Changes

Table 11. Changes

ID Type Summary


Add job report: show 'active' hosts,show more size totals


Add "Site-first"/'frontline' prioritiza-tion


Add multiple open http connectionsper host needed


Add Add help to web UI


86





ID Type Summary


Add When a host last had a completedURI shown in crawl report


Add Encode multibyte URIs usingpage charset before queuing


Add One src for site, help, and readmedocs.


Add identifying ARCs: unique names,header records


Add Resetting arc file counter forevery job.


Add ARCs need better headers


Add Specify location of jobs dir


Add Logging in (HTTP POST, BasicAuth, etc.)


Add Update dnsjava from 1.5 to 1.6.2(Fix NPE)

966168[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833amp;atid=539099amp;aid=966168]

Fix crawl.log entries without annota-tions end with a space

966172[http://sourceforge.net/tracker/ind

Fix An issue with arc names' date andserial number alignment


87










http://sourceforge.net/tracker/index.php?func=detail&group_id=73833amp;atid=539099amp;aid=966168


ID Type Summary

ex.php?func=detail&group_id=73833amp;atid=539099amp;aid=966172]


Fix Output of warning message leadsto NullPointerExceptions


Fix Either UURI or ExtractHTMLshould strip whitespace better


Fix Maximum documents not en-forced


Fix NPE in path depth filter


Fix embed/speculative inclusion tooloose


Fix UnsupportedCharsetExceptionhandled awkwardly


Fix UURI accepting/creating unUs-able URIs (bad hosts)


Fix CachingDiskLongFPSet UI avail-ability


Fix Crawls slow till change a setting

961867[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833amp;atid=539099amp;aid=9

Fix zero link-hops should work


88











ID Type Summary

61867]


Fix multiple robots.txt URLs in the"default" frontier


Fix NPE in Extract-orHTML#isHtmlExpectedHere


Fix Unwanted behavior with seed re-direction


Fix Link extraction failing


Fix Memory issues: Fronti-er.snoozeQueue


Fix Transitive scope confusion, maynot work as expected


Fix Wrong stats after deleting URIsfrom Frontier


Fix NoSuchElementException in ad-min/reports/frontier.jsp


Fix Alert: Authentication scheme(s)not supported


Fix IP validity: units, TTL vs. setting

951582 Fix ConcurrentModificationExcep-


89












ID Type Summary

[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833amp;atid=539099amp;aid=951582]

tion in DomainScope focus filter


Fix ConcurrentModificationExcep-tion terminate job


Fix Authentication bug


Fix terminate running crawl == NPE


Fix java.net.URI parses %20 butgetHost null


Fix NPE in java.net.URI.encode


Fix java.net.URI chokes onhosts_with_underscores


Fix Doing separate DNS lookup forsame host


Fix java.net.URI#getHost fails whenleading digit


Fix Constraining java URI class


Fix Same CrawlServer instance forhttp & https.


90











ID Type Summary

3833amp;atid=539099amp;aid=943373]


Fix Broad crawl/ too many open files


Fix multiple charset headers + longlines


Fix Corrupted blue image in progressbars


Fix NPEs in Andy's Th-Fri Crawl +NPE in RIS


Fix IllegalArgumentEx/Re-playCharSequenceFactory (offsetvs. size


Fix FTP URIs in seeds interpreted asHTTP


Fix maven rc2 won't make src distri-bution


Fix Corrupted arc files on termina-tion of job


Fix https exception:java.io.IOException: SSL failure


Fix Excessive ARCWriterPooltimeouts:


91











19. Release 0.10.0 - 2004-06-04Fixes to build with maven rc2+.

19.1. Changes

Table 12. Changes

ID Type Summary


Fix 080 doesn't build with mavenrc2+

20. Release 0.8.0 - 2004-05-18Release (and branch heritrix-0_8 made at the heritrix-0_7_1 tag) because of concurrentmodificationex-ceptions if tens of seeds supplied and to fix domain-scope leakage. Also, made continuous build public-ally available, incorporated integration selftest into build, made it a maven-build only (ant-build nolonger supported), added day/night configurations (refinements), ameliorated too-many-open files, ad-ded exploit of http-header content-type charset creating character streams, and heritrix now crawls sslsites. UI improvements include red start by bad configuration, precompilation, and delineation of ad-vanced settings. Many bug fixes.

20.1. Synopsis

20.2. Changes

Table 13. Changes

ID Type Summary


Add integrate selftest intocruisecontrol build


Add On reedit, red star by bad attrib-ute setting.


Add day/night configurations


92





ID Type Summary


Add UI should only write changedconfig


Add record of settings changes shouldbe kept


Add Only one build, not two


Add maven-only build rather than ant& maven


Add ARCWriter should use a pool ofopen files -- if it helps


Add Precompile UI pages


Add UI should be split into common/uncommon settings


Add UI web pages need to be more re-sponsive


Fix domain scope leakage


Fix ConcurrentModificationExcep-tions


Fix Too many open files


93












ID Type Summary

ex.php?func=detail&group_id=73833&atid=539099&aid=943768]


Fix ConcurrentModificationExcep-tions


Fix empty seeds-report.txt


Fix Doc. assumes bash. Allow tcsh/csh


Fix script heritrix.sh goes into infiniteloop


Fix heritrix.sh launch file path weird-ness


Fix ToeThreads hung in Extract-orHTML after Pause


Fix IllegalCharsetNameException:Windows-1256


Fix No doc-files/package.html injavadoc.


Fix embed-count sensitivity WRT re-directs, preconditions


Fix Refinement limits are not alwayssaved


94











ID Type Summary

]


Fix NPE exception in getMBean-Info(settings)


Fix Untried CrawlURIs should haveclear status code


Fix Thread underutilization in broadcrawls


Fix KeyedQueue showing EMPTYstatus, but the length is 1.


Fix NPE in XMLSettingsHand-ler.recursiveFindFiles()


Fix Failed DNS does not have inten-ded impact


Fix ftp URIs are retried


Fix Refetching of robots and/or DNSbroken


Fix NPE switching to 'expert' settingsin HEAD


Fix rss extractor

896775 Fix JS extractor clueless on relative


95












ID Type Summary


URIs


Fix converting URI's '\' into '/' charac-ter


Fix When going back to overrides,directory is gone


Fix shutdown.jsp unable to compile


Fix ARCWriterPool timeouts -- legit-imate?


Fix If one URI connect-fails, holdqueue, too


Fix Fetching simple URLs fails withS_CONNECT_FAILED (-2) er-ror


Fix seeds held back/poor breadthfirst?


Fix Parsing links found between es-caped quotes in JavaScript


Fix Does not extract applet URI cor-rectky


Fix links to likely-embed typesshould be treated as embeds


96











ID Type Summary

3833&atid=539099&aid=791481]


Fix Frontier.next() forceFetches willcause assertion error


Fix Flash link extractor causesOutOfMemory exceptions.


Fix Should be possible to resumefrom


Fix Heritrix ignores charset


Fix Max # of arcs not being respec-ted.


Fix New profile should ensure uniquename


Fix When changing scope commonscope settings are lost


Fix ssl doesn't work


Fix Allow that people use tcsh/cshnot just bash


Fix https SSLHandshakeException:unknown certificate


97











ID Type Summary


Fix Cannot override settings that isn'tset in globals


Fix 'Waiting for pause' even after allthreads done


Fix filter 'invert', filter names needwork


Fix max-link-hops (etc.) ignored un-less


Fix order.xml absolute paths


Fix Cannot set TransclusionFilter at-tributes


Fix Link puts garbage into arc file:http://www.msn.com/robots.t

21. Release 0.6.0 - 2004-03-25Release made in advance of radical frontier changes. Added bandwidth throttle, operator 'diary', settablerobots expiration, crawler cookie pre-population, and changing of certain options mid-crawl. Many UIimprovements including UI display of critical exceptions, UI desccription of job-order options, and im-proved reporting. Optimizations. Updated httpclient lib to 2.0 release and jmx libs to 1.2.1. Lots of bugfixes.

21.1. Changes

Table 14. Changes


98








ID Type Summary


Add 861861 Redirects(/refreshes)from seeds should == new seeds


Add 899223 Special seed-success re-port should be offered


Add 891986 Bandwidth throttle func-tion, setting.


Add 877275 integrated operator 'diary'needed


Add 891983 IP, Robots expirationsshould be settable


Add 910152 Recovery of old jobs onWUI (re)start


Add 781171 parsing css


Add 912986 log views should give anidea of file size (where possible)


Add 912989 Alerts should have 'selectall' button...


Add 856593 [load][save][turn on/off]cookies


Add 912201 Add levels to alerts


99












ID Type Summary

ex.php?func=detail&group_id=73833&atid=539099&aid=912201]


Add 896665 Split processor chains.


Add 896754 Show total of disregards


Add 903095 Show increments ofmegabytes in ui


Add 896794 serious errors (egoutofmemory) should show up inUI


Add 900520 Short description ofComplexTypes in user interface.


Add 899982 Should be possible to al-ter filters while crawling.


Add 896672 Display progress(doc/sec) with more precision


Add 896677 Highlight the success orfailures of each seed


Add 896760 Prominent notificationwhen seeds have problems


Add 896801 java regexps (in logview) need help text


100











ID Type Summary

]


Add 896778 Log viewing enhance-ments:


Add 896795 frontier, thread report im-provements


Add 876516 default launch should no-hup, save stdout/stderr


Fix 127.0.0.1 in job report


Fix Frontier retry-delay should in-clude units (eg -seconds)


Fix Revisiting admin URIs if notlogged in should prompt login


Fix Deadlock in Andy's 2nd Crawl


Fix Better bad-config handling


Fix mysterious pause facing network(DNS) problem


Fix ExtractorJS's report overstates it'sdiscovered URIs

896667 Fix Web UI does not display cor-


101












ID Type Summary


rectly in IE


Fix console clarity/safety


Fix Does not respect per settings ad-ded after crawl was started.


Fix 'empty' records in compressed arcfiles

22. Release 0.4.1 - 2004-06-04Memory retention fix.

22.1. Changes

Table 15. Changes

ID Type Summary

- Fix 895955[http://sourceforge.net/tracker/index.php?func=detail&aid=895955&group_id=73833&atid=539099]URIRegExpFilter retainsmemory

23. Release 0.4.0 - 2004-02-10Release made for heritrix workshop, San Francisco, 02/2004. New MBEAN-based configuration, ex-tensive UI revamp, first unit tests and integration selftest framework added, pooling of ARCWriters,new cmd-line start scripts, httpclient lib update (2.0RC3) and bugfixes.

23.1. Changes

Table 16. Feature


102





ID Type Summary

- Add New MBEAN-based configura-tion system. Reads and writesXML to validate against herit-rix_settings.xsd.

- Add UI extensively revamped. Ex-ploits new configuration system.

- Add 60-odd unit tests added.

- Add Integration selftest framework.

- Add Added pooling of ARCWriters.

- Add Start script backgrounds heritrixand redirects stdout/stderr to

- Add 876516[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=876516] Default launch should nohup,save stdout/stderr

- Add Web UI accesses are loggged toheritrix_out.log also.

- Add Updated httpclient to version2.0RC3.

- Add 763517[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=763517] IAGZIPOutputStream NPE un-der IBM JVM

- Add 809018[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=809018] Cleaner versioned testing buildneeded

- Add 872729[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=872729] Cmd-line options for settingweb ui username/password

- Add 863317[http://sourceforge.net/tracker/index.php?func=detail&group_id=73833&atid=539099&aid=863317] Universal single-pass extractor

24. Release 0.2.0 - 2004-01-05First 'official' release.


103






25. Release 0.1.0 - 2003-12-31Initial Mavenized development version number (CVS/internal only). Added everything to new projectlayout.


104

heritrix release notes · 6. release 1.12.1 - 2007-05-06 release 1.12.1 is primarily a bug-fix...

Documents