You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 47 Next »

Under construction

Note on   from Lewis McGibbney 

This page is under construction.

Introduction

This page provides a narrative on Nutch application metrics. It details which metrics are captured for which Nutch Job's within which Tasks.

Metrics are important because they tell you vital information about any given Nutch (and subsequently MapReduce) process. They provide accurate measurements about how the process is functioning and provide basis to suggest improvements.

Metrics provide a data-driven mechanism for intelligence gathering within Nutch operations and administration.

Audience

The page is intended for

  • users who wish to learn about how Nutch Jobs and Tasks are performing, and
  • developers who would wish to further extend/customize Nutch metrics

Related Development Work

  • Unable to render Jira issues macro, execution error.
  • Several rows in the metrics table below reference JIRA issues.

Building Metrics on MapReduce Context's

As Nutch is a native MapReduce application, the Mapper and Reducer functions of each NutchTool implementation i.e. CommonCrawlDataDumper, CrawlDb, DeduplicationJob, Fetcher, Generator, IndexingJob, Injector, LinkDb, ParseSegment utilize MapContext's and ReduceContext's. These Context's are passed to the Mapper and Reducer initially during setup but also used throughout each Mapper or Reducer task lifecycle.

Hadoop documentation

The canonical Hadoop documentation for Mapper and Reducer provides much more detail about the involvement of Context's in each task lifecycle.

This is relevant because these Context's inherit certain methods from the interface org.apache.hadoop.mapreduce.TaskAttemptContext. Specifically, the getCounter(...) methods facilitate access to Hadoop Counter's which we discuss below.

Hadoop Counter's

A Counter is simply a record comprising a name and value. As one would expect, Counter's can be incremented in order to count for example how many total records were processed within a task completion.

The following example shows how Counter's are used within the Nutch Injector to count the total number of URLs filtered during the Map phase of this job.

Use of Counters in the Nutch Injector 
    @Override
    public void map(Text key, Writable value, Context context)
        throws IOException, InterruptedException {
      if (value instanceof Text) {
        // if its a url from the seed list
        String url = key.toString().trim();

        // remove empty string or string starting with '#'
        if (url.length() == 0 || url.startsWith("#"))
          return;

        url = filterNormalize(url);
        if (url == null) {
          context.getCounter("injector", "urls_filtered").increment(1);

The code on Line 14 demonstrates the urls_filtered counter for injector counter group being incremented by 1.

The end result is that we generate useful, insightful metrics for each mapper and reducer task for any given Nutch Job.

See below for details on each Nutch metric available.

Metrics Table

The table below provides a canonical, comprehensive collection of Nutch metrics.

Table Ordering Logic

The table is arranged

  1. by Tool column; alphabetically
  2. by the Metric Group; alphabetically for the given tool
  3. by Metric Name; alphabetically for the given metric group
Tool/ObjectMetric GroupMetric NameDescriptionUsage and Comments
CleaningJobCleaningJobStatusDeleted documentsThe total count of DB_GONE (HTTP 404) and/or DB_DUPLICATE documents ultimately deleted from the indexer(s).This metric is useful for determining whether filtering or duplicate detection needs to happen further upstream prior to indexing. Ideally DB_GONE and DB_DUPLICATE documents would not make it into production indices in the first place.
CrawlDbFilterCrawlDB filterGone records removedThe total count of DB_GONE (HTTP 404) records deleted from the CrawlDB during an update.

See Unable to render Jira issues macro, execution error. for more details.

CrawlDB filterOrphan records removedThe total count of orphaned pages e.g. a page which have no more other pages linking to it, deleted from the CrawlDB during an update.

See  Unable to render Jira issues macro, execution error. for more details.

CrawlDB filterURLs filteredThe total count of filtered pages e.g. pages which didn't pass one or more URLFIlter implementation(s), deleted from the CrawlDB during an update.

This metric is generally useful for determining the overall effectiveness of URLFilter plugins over time.

POSSIBLE IMPROVEMENT: This metric could be improved if an association could be made between the URL was filtered and the URLFilter which filtered it. This would facilitate aggregating URLFiltering results by URLFilter.

CrawlDbReducerCrawlDB status

CrawlDatum.getStatusName(CrawlDatum().getStatus())

With each URL able to have only one state at any given point in time, this metric facilitates aggregated counts of the different types of CrawlDatum states for a given CrawlDB.

The state of any given URL will change as the URL transitions through a crawl cycle. Available URL states are defined in the CrawlDatum e.g., STATUS_DB_UNFETCHED, STATUS_DB_FETCHED, STATUS_FETCH_SUCCESS, etc. Practically, CrawlDatum status' are defined using byte signatures but accessed programmatically using static final constants.

This metric can be used to identify the presence of undesired URL CrawlDatum status' for given URL's e.g., STATUS_DB_GONE. Such an event could then trigger a cleaning/pruning operation.

DeduplicationJobDeduplicationJobStatusDocuments marked as duplicateThe total number of duplicate documents in the CrawlDB.

The process of identifying (near) duplicate documents is of vital importance within the context of a search engine. The precision of any given information retrieval system can be negatively impacted if (near) duplicates are not identified and handled correctly. This does not always mean removing them, for example maybe (near) duplicates are important for versioning purposes. In most cases however it is preferred to identify and remove (near) duplicate records.

The Deduplication algorithm in Nutch groups fetched URLs with the same digest and marks all of them as duplicate except the one with the highest score (based on the score in the crawldb, which is not necessarily the same as the score indexed). If two (or more) documents have the same score, then the document with the latest timestamp is kept. If the documents have the same timestamp then the one with the shortest URL is kept.

A duplicate record will have a CrawlDatum status of CrawlDatum.STATUS_DB_DUPLICATE.

DomainStatistics

N/AMyCounter.EMPTY_RESULTThe total count of empty (probably problematic) URL records for a given host, domain, suffix or top-level domain.It is possible that the DomainStatistics tool may identify an empty record for a given URL. This may happen regardless of whether the tool is invoked to retrieve host, domain, suffix or top-level domain statistics. When this discovery event occurs, it it is likely that some investigation would take place to understand why. For example, the CrawlDbReader could be invoked with the -url command line argument to further debug/detail what CrawlDatum data exists.
N/AMyCounter.FETCHEDThe total count of fetched URL records for a given host, domain, suffix or top-level domain.This metric is particularly useful for quickly drilling down through large datasets to determine, for example, how much 'coverage' has been achieved for a given host, domain, suffix or top-level domain. This figure can be compared to a website administrators total.
N/AMyCounter.NOT_FETCHEDThe total count of unfetched URL records for a given host, domain, suffix or top-level domain.This metric is particularly useful for quickly drilling down through large datasets to determine, for example, how much 'coverage' still has to be achieved for a given host, domain, suffix or top-level domain. When combined with the fetched figure and compared to a website administrators total it can provide useful insight.

Fetcher		FetcherStatusbytes_downloadedThe total bytes of fetched data acquired across the Fetcher Mapper task(s).

Over time, this can be used to benchmark how much data movement is occurring over the Nutch crawl network.

POSSIBLE IMPROVEMENT: This metric could be improved if a correlation could be made between the volume of data and the source it came from whether that be a given host, domain, suffix or top-level domain.

FetcherStatushitByThrougputThresholdA total count of the URLs dropped across all fetch queues due to throughput dropping below the threshold too many times.

This aspect of the Nutch Fetcher configuration is designed to prevent slow fetch queues from stalling the overall fetcher throughput. However it usually has the impact of increasing the latency/timeliness of URLs actually being fetched if they are essentially dropped because of low throughput. This means that they are shelved until a future fetch operation.

The specific configuration settings is

fetcher.throughput.threshold.pages 
<property>
  <name>fetcher.throughput.threshold.pages</name>
  <value>-1</value>
  <description>The threshold of minimum pages per second. If the fetcher downloads less
  pages per second than the configured threshold, the fetcher stops, preventing slow queue's
  from stalling the throughput. This threshold must be an integer. This can be useful when
  fetcher.timelimit.mins is hard to determine. The default value of -1 disables this check.
  </description>
</property>

A more thorough understanding of Fetcher configuration relating to (slow) throughput requires an understanding of the following configuration settings as well

Additional fetcher throughput configuration 
<property>
  <name>fetcher.throughput.threshold.retries</name>
  <value>5</value>
  <description>The number of times the fetcher.throughput.threshold.pages is allowed to be exceeded.
  This settings prevents accidental slow downs from immediately killing the fetcher thread.
  </description>
</property>

<property>
<name>fetcher.throughput.threshold.check.after</name>
  <value>5</value>
  <description>The number of minutes after which the throughput check is enabled.</description>
</property>

POSSIBLE IMPROVEMENT: It would be advantageous to understand which URLs from which hosts in the queue(s) were resulting in slow throughput. This would facilitate investigation into why this was happening.

FetcherStatushitByTimeLimitA total count of the URLs dropped across all fetch queues due to the fetcher execution time limit being exceeded.

This metric is valuable for quantifying the number of URLs which have been effectively timebombed e.g. shelved for future fetching due to overall fetcher runtime exceeding a predefined timeout.

Although by default the Fetcher never times out e.g. the configuration is set to -1, if a timeout is preferred then the following configuration property can be edited.

fetcher.timelimit.mins 
<property>
  <name>fetcher.timelimit.mins</name>
  <value>-1</value>
  <description>This is the number of minutes allocated to the fetching.
  Once this value is reached, any remaining entry from the input URL list is skipped 
  and all active queues are emptied. The default value of -1 deactivates the time limit.
  </description>
</property>

POSSIBLE IMPROVEMENT: It could be useful to record the fact that a URL was staged due to it being hit by the timeout limit. This could possibly be stored in the CrawlDatum metadata.

Also see Unable to render Jira issues macro, execution error.









FetcherThread		FetcherStatusAboveExceptionThresholdInQueueThe total count of URLs purged across all fetch queues as a result of the maximum number of protocol-level exceptions (e.g. timeouts) per queue being exceeded.

This metric is useful for quantifying the number of URLs shelved for future fetching due to anomalies occurring during fetcher execution exceeding a predefined ceiling.

Although by default the Fetcher never enforces this behaviour e.g. the configuration is set to -1, if this is changed then this total count will become a useful metric to track. Further information on the configuration parameter can be seen below

fetcher.max.exceptions.per.queue 
<property>
  <name>fetcher.max.exceptions.per.queue</name>
  <value>-1</value>
  <description>The maximum number of protocol-level exceptions (e.g. timeouts) per
  host (or IP) queue. Once this value is reached, any remaining entries from this
  queue are purged, effectively stopping the fetching from this host/IP. The default
  value of -1 deactivates this limit.
  </description>
</property>

POSSIBLE IMPROVEMENT: It could be useful to record the fact that a URL was staged due to it being hit by the exception limit. Additionally, it could be useful to write metadata to all URLs which contributed towards the limit being met. This could possibly be stored in the CrawlDatum metadata.

FetcherStatusFetchItem.notCreated.redirectA total count of URLs across all fetch queues for which following redirect(s) resulted in no result.Essentially, each FetcherThread attempts to understand and eliminate all redirect options e.g. duplicate redirect URL, before giving up on a redirect URL entirely. In the case of a redirect URL for which no logical fetch outcome can be produced e.g. that the FetchItem is null, redirecting is simply deactivated as it is impossible to continue.
FetcherStatusoutlinks_detectedA total count of detected outlinks for all fetch items (URLs) across all fetch queues.This metric can be used to estimate the number in URLs to be fetched in the next fetch phase. If for example, resources were being allocated within the client at configuration/compile time (rather than dynamically at runtime) this could be used to inform resource reservations, utilization and data partitioning logic.
FetcherStatusoutlinks_followingFrom outlinks_detected (see directly above), this is a total count of URLs which will be followed.

This metric value may be the same as outlinks_detected or it may be less. This ultimately depends on a few things i.e.,

  • whether the fetcher is configured to follow external outlinks
  • whether a given URL is already followed
  • whether a given URL is already fetched
FetcherStatusProtocolStatus.getName()Total counts of all the different fetched finished status' for all URLs.For a comprehensive collection of the various fetched finish status' to expect, check out the private static final HashMap<Integer, String> codeToName defined within ProtocolStatus. This metric is useful for understanding, from across your CrawlDb, the status of certain URLs. Via different tools, you can begin to investigate further. 
FetcherStatusredirect_count_exceededTotal count of all URLs which have exceeded the maximum configured number of redirects.

See the following configuration property in nutch-default.xml

http.redirect.max 
<property>
  <name>http.redirect.max</name>
  <value>0</value>
  <description>The maximum number of redirects the fetcher will follow when
  trying to fetch a page. If set to negative or 0, fetcher won't immediately
  follow redirected URLs, instead it will record them for later fetching.
  </description>
</property>

This metric is useful for understanding how many URLs are possibly skipped due to a large number of redirects.

Also see the following property

http.redirect.max.exceeded.skip 
<property>
  <name>http.redirect.max.exceeded.skip</name>
  <value>false</value>
  <description>
    Whether to skip the last URL in a redirect chain when when redirects
    are followed (http.redirect.max > 0) and the maximum number of redirects
    in a chain is exceeded (redirect_count > http.redirect.max).
    If not skipped the redirect target URLs are stored as `linked`
    and fetched in one of the following cycles. See also NUTCH-2748.
  </description>
</property>
FetcherStatusredirect_deduplicatedTotal count of duplicate (and hence not fetched) redirected URLs.No fetching takes place for this class of redirect URLs as they are duplicates of other redirect URLs already fetched.
FetcherStatusrobots_deniedTotal count of all URLs not fetched due to being denied by robots.txt rules.By default Nutch is configured to respect and comply with robots.txt rules for any given site. It is useful to know how many URLs may not be fetched from a given site due to robots.txt compliance.
FetcherStatusrobots_denied_maxcrawldelayTotal count of URLs which are skipped due to the robots.txt crawl delay being above a configured maximum.

The following configuration property must be consulted for a detailed explanation behind this metric.

fetcher.max.crawl.delay 
<property>
 <name>fetcher.max.crawl.delay</name>
 <value>30</value>
 <description>
 If the Crawl-Delay in robots.txt is set to greater than this value (in
 seconds) then the fetcher will skip this page, generating an error report.
 If set to -1 the fetcher will never skip such pages and will wait the
 amount of time retrieved from robots.txt Crawl-Delay, however long that
 might be.
 </description>
</property>

Essentially, a delay of 5 seconds is used for fetching requests to the same host unless a crawl delay is specified within the robots.txt. Also see

fetcher.server.delay 
<property>
  <name>fetcher.server.delay</name>
  <value>5.0</value>
  <description>The number of seconds the fetcher will delay between 
   successive requests to the same server. Note that this might get
   overridden by a Crawl-Delay from a robots.txt and is used ONLY if 
   fetcher.threads.per.queue is set to 1.
   </description>
</property>
ParserStatusParseStatus.majorCodes[p.getData().getStatus().getMajorCode()]Total count of  major codes (see right) from parsing URLs.ParseStatus defines three major categories for the result of an URL parse operation. i.e., notparsed, success and failed. This metric is useful for debugging how many parse operations failed for a given crawl cycle. Subsequent parse attempts can then be made or the URL record can be handled appropriately.







Generator















GeneratorEXPR_REJECTEDTotal count of URLs rejected by Jexl expression(s) during the generate phase.

This metric is useful for determining the impact that that Jexl expressions (provided via the Generator CLI) have on filtering URLs. The expressions are evaluated during the Generator Map phase.

All of the configuration which drives this metric is read or set from Java code and not explicitly defined in nutch-default.xml.

GeneratorHOSTS_AFFECTED_PER_HOST_OVERFLOWTotal count of host(s) or domain(s) affected by the number of URLs exceeding a configured fetchlist size threshold.

This configuration property is essentially turned off by default e.g. there is no defined maximum number of URLs per fetchlist. However, if a maximum is defined, then it is useful to know, how many hosts or domains included in fetchlists have more URLs than allowed. In these cases additional URLs won't be included in the fetchlist but bumped on to future crawling cycles.

The configuration property below will directly drive this metric.

generate.max.count 
<property>
  <name>generate.max.count</name>
  <value>-1</value>
  <description>The maximum number of URLs in a single
  fetchlist. -1 if unlimited. The URLs are counted according
  to the value of the parameter generate.count.mode.
  </description>
</property>
GeneratorINTERVAL_REJECTEDTotal count of records rejected due to retry or fetch interval being above a configured thershold.

This configuration property is essentially turned off by default e.g. there is no minimum defined retry interval. This metric is useful for understanding the impact that changing that has on URL filtering.

The configuration property below drives this metric.

generate.min.interval 
<property>
  <name>generate.min.interval</name>
  <value>-1</value>
  <description>Select only entries with a retry interval lower than
  generate.min.interval. A value of -1 disables this check.</description>
</property>
GeneratorMALFORMED_URLTotal count of malformed URLs filtered.

In the Generator, malformed URLs are either discovered by

  1. an URL normalizer implementation. This is turned on by default but can be toggled on or off within the Generator CLI or programmatically
  2. Attempting to extract either the URL host or domain (depending on which one is configured). See the following property
generate.count.mode 
<property>
  <name>generate.count.mode</name>
  <value>host</value>
  <description>Determines how the URLs are counted for generate.max.count.
  Default value is 'host' but can be 'domain'. Note that we do not count 
  per IP in the new version of the Generator.
  </description>
</property>
GeneratorSCHEDULE_REJECTEDTotal count of URLs not suitable for selection (in a given crawl cycle due to the fetch time being  higher than the current time.The metric description covers the default fetch schedule case but this can change depending on the actual implementation. Of specific interest is FetchSchedule#shouldFetch(...) which explains further. This metric can be useful for comparing implementations of FetchSchedule.
GeneratorSCORE_TOO_LOWTotal count of filtered URL entries with a score lower than a configured threshold.

The configuration parameter which drives this metric is

generate.min.score 
<property>
  <name>generate.min.score</name>
  <value>0</value>
  <description>Select only entries with a score larger than
  generate.min.score.</description>
</property>

The default value for this configuration property means that all entries should be selected.

The metric can be useful to determine if a configured minimum value is too high and filters too many URLs from being included in fetchlists.

GeneratorSTATUS_REJECTEDTotal count of URLs filtered by a  CrawlDatum status filter.

The following configuration property is used to straight filter URLs depending on their CrawlDatum status

generate.restrict.status 
<property>
  <name>generate.restrict.status</name>
  <value></value>
  <description>Select only entries of this status, see
  https://issues.apache.org/jira/browse/NUTCH-1248</description>
</property>

As an indication of the status keys which can be used, see CrawlDatum.statNames.

This metric is useful to simply see how effective the status filters are.

GeneratorURLS_SKIPPED_PER_HOST_OVERFLOWTotal count of URLs skipped by the number of URLs exceeding a configured fetchlist size threshold.

This configuration property is essentially turned off by default e.g. there is no defined maximum number of URLs per fetchlist. However, if a maximum is defined, then it is useful to know, how many URLs are skipped. In these cases additional URLs won't be included in the fetchlist but bumped on to future crawling cycles.

The configuration property below will directly drive this metric.

generate.max.count 
<property>
  <name>generate.max.count</name>
  <value>-1</value>
  <description>The maximum number of URLs in a single
  fetchlist. -1 if unlimited. The URLs are counted according
  to the value of the parameter generate.count.mode.
  </description>
</property>
IndexerMapReduce








IndexerStatusdeleted (duplicates)

IndexerStatusdeleted (IndexingFilter)

IndexerStatusdeleted (gone)

IndexerStatusdeleted (redirects)

IndexerStatusdeleted (robots=noindex)

IndexerStatuserrors (IndexingFilter)

IndexerStatuserrors (ScoringFilter)

IndexerStatusindexed (add/update)

IndexerStatusskipped (IndexingFilter)

IndexerStatusskipped (not modified)





Injector



injectorurls_filteredTotal count of seed URLs filtered by the Injector.

URL normalization and then filtering operations are executed within the Injector Map task(s). They are both turned off by default in nutch-default.xml however if these values are not interpreted the Injector turns normalization and filtering operations on by default.

This metric is useful to determine the impact that normalization and filtering have on the injection of seeds contained within seed lists. For more information on configuration see

Normalization and Filtering 
<property>
    <name>crawldb.url.normalizers</name>
    <value>false</value>
    <description>
	!Temporary, can be overwritten with the command line!
	Normalize URLs when updating crawldb
    </description>
</property>

<property>
    <name>crawldb.url.filters</name>
    <value>false</value>
    <description>
	!Temporary, can be overwritten with the command line!
	Filter URLS when updating crawldb
    </description>
</property>
injectorurls_injectedTotal count of seed URLs injected by the Injector.A useful metric for simply counting by how many URLs the CrawlDb has grown by the end of an Injector invocation.
injectorurls_mergedTotal count of seed URLs merged with an existing CrawlDatum record.
injectorurls_purged_404Total count of deleted/purged URLs due to an existing CrawlDatum.STATUS_DB_GONE
injectorurls_purged_filterTotal count of deleted/purged URLs filtered by one or more filters and/or normalizers.
ParseSegmentParserStatusParseStatus.majorCodes[parseStatus.getMajorCode()]

QueueFeederFetcherStatusfiltered

(also QueueFeeder)FetcherStatusAboveExceptionThresholdInQueue

ResolverThread






UpdateHostDbchecked_hosts

UpdateHostDbexisting_known_host

UpdateHostDbexisting_unknown_host

UpdateHostDbnew_known_host

UpdateHostDbnew_unknown_host

UpdateHostDbpurged_unknown_host

UpdateHostDbrediscovered_host

UpdateHostDbLong.toString(datum.numFailures()) + "_times_failed"

SitemapProcessor





Sitemapexisting_sitemap_entries

Sitemapfailed_fetches

Sitemapfiltered_records

Sitemapfiltered_sitemaps_from_hostname

Sitemapnew_sitemap_entries

Sitemapsitemaps_from_hostname

Sitemapsitemap_seeds

UpdateHostDbMapperUpdateHostDbfiltered_records

UpdateHostDbReducerUpdateHostDbtotal_hosts

(also UpdateHostDbReducer)UpdateHostDbskipped_not_eligible

WebGraphWebGraph.outlinksadded links

(also WebGraph)WebGraph.outlinksremoved links

WARCExporter




WARCExporterexception

WARCExporterinvalid URI

WARCExportermissing content

WARCExportermissing metadata

WARCExporteromitted empty response

WARCExporterrecords generated

Conclusion

This document aims to provide a detailed account of Nutch application metrics such that a data-driven approach can be adopted to better manage Nutch operations. Several cells in the USAGE AND COMMENTS column in the above table offer areas for POSSIBLE IMPROVEMENT. These suggestions are targeted towards Nutch crawler administrators and developers interested in possibly improving Nutch metrics.

  • No labels