You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 31 Next »

Under construction

Note on   from Lewis McGibbney 

This page is under construction.

Introduction

This page provides a narrative on Nutch application metrics. It details which metrics are captured for which Nutch Job's within which Tasks.

Metrics are important because they tell you vital information about any given Nutch (and subsequently MapReduce) process. They provide accurate measurements about how the process is functioning and provide basis to suggest improvements.

Metrics provide a data-driven mechanism for intelligence gathering within Nutch operations and administration.

Audience

The page is intended for

  • users who wish to learn about how Nutch Jobs and Tasks are performing, and
  • developers who would wish to further extend/customize Nutch metrics

Related Development Work

  • Unable to render Jira issues macro, execution error.

Building Metrics on MapReduce Context's

As Nutch is a native MapReduce application, the Mapper and Reducer functions of each NutchTool implementation i.e. CommonCrawlDataDumper, CrawlDb, DeduplicationJob, Fetcher, Generator, IndexingJob, Injector, LinkDb, ParseSegment utilize MapContext's and ReduceContext's. These Context's are passed to the Mapper and Reducer initially during setup but also used throughout each Mapper or Reducer task lifecycle.

Hadoop documentation

The canonical Hadoop documentation for Mapper and Reducer provides much more detail about the involvement of Context's in each task lifecycle.

This is relevant because these Context's inherit certain methods from the interface org.apache.hadoop.mapreduce.TaskAttemptContext. Specifically, the getCounter(...) methods facilitate access to Hadoop Counter's which we discuss below.

Hadoop Counter's

A Counter is simply a record comprising a name and value. As one would expect, Counter's can be incremented in order to count for example how many total records were processed within a task completion.

The following example shows how Counter's are used within the Nutch Injector to count the total number of URLs filtered during the Map phase of this job.

Use of Counters in the Nutch Injector
    @Override
    public void map(Text key, Writable value, Context context)
        throws IOException, InterruptedException {
      if (value instanceof Text) {
        // if its a url from the seed list
        String url = key.toString().trim();

        // remove empty string or string starting with '#'
        if (url.length() == 0 || url.startsWith("#"))
          return;

        url = filterNormalize(url);
        if (url == null) {
          context.getCounter("injector", "urls_filtered").increment(1);

The code on Line 14 demonstrates the urls_filtered counter for injector counter group being incremented by 1.

The end result is that we generate useful, insightful metrics for each mapper and reducer task for any given Nutch Job.

See below for details on each Nutch metric available.

Metrics Table

The table below provides a canonical, comprehensive collection of Nutch metrics.

Table Ordering Logic

The table is arranged

  1. by Tool column; alphabetically
  2. by the Metric Group; alphabetically for the given tool
  3. by Metric Name; alphabetically for the given metric group


Tool/ObjectMetric GroupMetric NameDescriptionUsage and Comments
CleaningJobCleaningJobStatusDeleted documentsThe total count of DB_GONE (HTTP 404) and/or DB_DUPLICATE documents ultimately deleted from the indexer(s).This metric is useful for determining whether filtering or duplicate detection needs to happen further upstream prior to indexing. Ideally DB_GONE and DB_DUPLICATE documents would not make it into production indices in the first place.
CrawlDbFilterCrawlDB filterGone records removedThe total count of DB_GONE (HTTP 404) records deleted from the CrawlDB during an update.

See Unable to render Jira issues macro, execution error. for more details.

CrawlDB filterOrphan records removedThe total count of orphaned pages e.g. a page which have no more other pages linking to it, deleted from the CrawlDB during an update.

See  Unable to render Jira issues macro, execution error. for more details.

CrawlDB filterURLs filteredThe total count of filtered pages e.g. pages which didn't pass one or more URLFIlter implementation(s), deleted from the CrawlDB during an update.

This metric is generally useful for determining the overall effectiveness of URLFilter plugins over time.

This metric could be improved if an association could be made between the URL was filtered and the URLFilter which filtered it. This would facilitate aggregating URLFiltering results by URLFilter.

CrawlDbReducerCrawlDB status

CrawlDatum.getStatusName(CrawlDatum().getStatus())

With each URL able to have only one state at any given point in time, this metric facilitates aggregated counts of the different types of CrawlDatum states for a given CrawlDB.

The state of any given URL will change as the URL transitions through a crawl cycle. Available URL states are defined in the CrawlDatum e.g., STATUS_DB_UNFETCHED, STATUS_DB_FETCHED, STATUS_FETCH_SUCCESS, etc. Practically, CrawlDatum status' are defined using byte signatures but accessed programmatically using static final constants.

This metric can be used to identify the presence of undesired URL CrawlDatum status' for given URL's e.g., STATUS_DB_GONE. Such an event could then trigger a cleaning/pruning operation.

DeduplicationJobDeduplicationJobStatusDocuments marked as duplicateThe total number of duplicate documents in the CrawlDB.

The process of identifying (near) duplicate documents is of vital importance within the context of a search engine. The precision of any given information retrieval system can be negatively impacted if (near) duplicates are not identified and handled correctly. This does not always mean removing them, for example maybe (near) duplicates are important for versioning purposes. In most cases however it is preferred to identify and remove (near) duplicate records.

The Deduplication algorithm in Nutch groups fetched URLs with the same digest and marks all of them as duplicate except the one with the highest score (based on the score in the crawldb, which is not necessarily the same as the score indexed). If two (or more) documents have the same score, then the document with the latest timestamp is kept. If the documents have the same timestamp then the one with the shortest URL is kept.

A duplicate record will have a CrawlDatum status of CrawlDatum.STATUS_DB_DUPLICATE.

DomainStatistics

N/AMyCounter.EMPTY_RESULT

N/AMyCounter.FETCHED

N/AMyCounter.NOT_FETCHED


Fetcher
FetcherStatusbytes_downloaded

FetcherStatushitByThrougputThreshold

FetcherStatushitByTimeLimit









FetcherThread
FetcherStatusAboveExceptionThresholdInQueue

FetcherStatusFetchItem.notCreated.redirect

FetcherStatusoutlinks_detected

FetcherStatusoutlinks_following

FetcherStatusProtocolStatus.getName()

FetcherStatusredirect_count_exceeded

FetcherStatusredirect_deduplicated

FetcherStatusrobots_denied

FetcherStatusrobots_denied_maxcrawldelay

ParserStatusParseStatus.majorCodes[p.getData().getStatus().getMajorCode()]








Generator















GeneratorEXPR_REJECTED

GeneratorHOSTS_AFFECTED_PER_HOST_OVERFLOW

GeneratorINTERVAL_REJECTED

GeneratorMALFORMED_URL

GeneratorSCHEDULE_REJECTED

GeneratorSCORE_TOO_LOW

GeneratorSTATUS_REJECTED

GeneratorURLS_SKIPPED_PER_HOST_OVERFLOW

IndexerMapReduce








IndexerStatusdeleted (duplicates)

IndexerStatusdeleted (IndexingFilter)

IndexerStatusdeleted (gone)

IndexerStatusdeleted (redirects)

IndexerStatusdeleted (robots=noindex)

IndexerStatuserrors (IndexingFilter)

IndexerStatuserrors (ScoringFilter)

IndexerStatusindexed (add/update)

IndexerStatusskipped (IndexingFilter)

IndexerStatusskipped (not modified)





Injector



injectorurls_filtered

injectorurls_injected

injectorurls_merged

injectorurls_purged_404

injectorurls_purged_filter

ParseSegmentParserStatusParseStatus.majorCodes[parseStatus.getMajorCode()]

QueueFeederFetcherStatusfiltered

(also QueueFeeder)FetcherStatusAboveExceptionThresholdInQueue

ResolverThread






UpdateHostDbchecked_hosts

UpdateHostDbexisting_known_host

UpdateHostDbexisting_unknown_host

UpdateHostDbnew_known_host

UpdateHostDbnew_unknown_host

UpdateHostDbpurged_unknown_host

UpdateHostDbrediscovered_host

UpdateHostDbLong.toString(datum.numFailures()) + "_times_failed"

SitemapProcessor





Sitemapexisting_sitemap_entries

Sitemapfailed_fetches

Sitemapfiltered_records

Sitemapfiltered_sitemaps_from_hostname

Sitemapnew_sitemap_entries

Sitemapsitemaps_from_hostname

Sitemapsitemap_seeds

UpdateHostDbMapperUpdateHostDbfiltered_records

UpdateHostDbReducerUpdateHostDbtotal_hosts

(also UpdateHostDbReducer)UpdateHostDbskipped_not_eligible

WebGraphWebGraph.outlinksadded links

(also WebGraph)WebGraph.outlinksremoved links

WARCExporter




WARCExporterexception

WARCExporterinvalid URI

WARCExportermissing content

WARCExportermissing metadata

WARCExporteromitted empty response

WARCExporterrecords generated

Conclusion

  • No labels