...
Tool/Object | Metric Group | Metric Name | Description | Usage and Comments | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CleaningJob | CleaningJobStatus | Deleted documents | The total count of DB_GONE (HTTP 404) and/or DB_DUPLICATE documents ultimately deleted from the indexer(s). | This metric is useful for determining whether filtering or duplicate detection needs to happen further upstream prior to indexing. Ideally DB_GONE and DB_DUPLICATE documents would not make it into production indices in the first place. | |||||||||||||||||||||
CrawlDbFilter | CrawlDB filter | Gone records removed | The total count of DB_GONE (HTTP 404) records deleted from the CrawlDB during an update. | See
| |||||||||||||||||||||
CrawlDB filter | Orphan records removed | The total count of orphaned pages e.g. a page which have no more other pages linking to it, deleted from the CrawlDB during an update. | See
| ||||||||||||||||||||||
CrawlDB filter | URLs filtered | The total count of filtered pages e.g. pages which didn't pass one or more URLFIlter implementation(s), deleted from the CrawlDB during an update. | This metric is generally useful for determining the overall effectiveness of URLFilter plugins over time. POSSIBLE IMPROVEMENT: This metric could be improved if an association could be made between the URL was filtered and the URLFilter which filtered it. This would facilitate aggregating URLFiltering results by URLFilter. | ||||||||||||||||||||||
CrawlDbReducer | CrawlDB status | CrawlDatum.getStatusName(CrawlDatum().getStatus()) | With each URL able to have only one state at any given point in time, this metric facilitates aggregated counts of the different types of CrawlDatum states for a given CrawlDB. | The state of any given URL will change as the URL transitions through a crawl cycle. Available URL states are defined in the CrawlDatum e.g., STATUS_DB_UNFETCHED, STATUS_DB_FETCHED, STATUS_FETCH_SUCCESS, etc. Practically, CrawlDatum status' are defined using byte signatures but accessed programmatically using static final constants. This metric can be used to identify the presence of undesired URL CrawlDatum status' for given URL's e.g., STATUS_DB_GONE. Such an event could then trigger a cleaning/pruning operation. | |||||||||||||||||||||
DeduplicationJob | DeduplicationJobStatus | Documents marked as duplicate | The total number of duplicate documents in the CrawlDB. | The process of identifying (near) duplicate documents is of vital importance within the context of a search engine. The precision of any given information retrieval system can be negatively impacted if (near) duplicates are not identified and handled correctly. This does not always mean removing them, for example maybe (near) duplicates are important for versioning purposes. In most cases however it is preferred to identify and remove (near) duplicate records. The Deduplication algorithm in Nutch groups fetched URLs with the same digest and marks all of them as duplicate except the one with the highest score (based on the score in the crawldb, which is not necessarily the same as the score indexed). If two (or more) documents have the same score, then the document with the latest timestamp is kept. If the documents have the same timestamp then the one with the shortest URL is kept. A duplicate record will have a CrawlDatum status of CrawlDatum.STATUS_DB_DUPLICATE. | |||||||||||||||||||||
DomainStatistics | N/A | MyCounter.EMPTY_RESULT | The total count of empty (probably problematic) URL records for a given host, domain, suffix or top-level domain. | It is possible that the DomainStatistics tool may identify an empty record for a given URL. This may happen regardless of whether the tool is invoked to retrieve host, domain, suffix or top-level domain statistics. When this discovery event occurs, it it is likely that some investigation would take place to understand why. For example, the CrawlDbReader could be invoked with the -url command line argument to further debug/detail what CrawlDatum data exists. | |||||||||||||||||||||
N/A | MyCounter.FETCHED | The total count of fetched URL records for a given host, domain, suffix or top-level domain. | This metric is particularly useful for quickly drilling down through large datasets to determine, for example, how much 'coverage' has been achieved for a given host, domain, suffix or top-level domain. This figure can be compared to a website administrators total. | ||||||||||||||||||||||
N/A | MyCounter.NOT_FETCHED | The total count of unfetched URL records for a given host, domain, suffix or top-level domain. | This metric is particularly useful for quickly drilling down through large datasets to determine, for example, how much 'coverage' still has to be achieved for a given host, domain, suffix or top-level domain. When combined with the fetched figure and compared to a website administrators total it can provide useful insight. | ||||||||||||||||||||||
Fetcher | FetcherStatus | bytes_downloaded | The total bytes of fetched data acquired across the Fetcher Mapper task(s). | Over time, this can be used to benchmark how much data movement is occurring over the Nutch crawl network. POSSIBLE IMPROVEMENT: This metric could be improved if a correlation could be made between the volume of data and the source it came from whether that be a given host, domain, suffix or top-level domain. | |||||||||||||||||||||
FetcherStatus | hitByThrougputThreshold | A total count of the URLs dropped across all fetch queues due to throughput dropping below the threshold too many times. | This aspect of the Nutch Fetcher configuration is designed to prevent slow fetch queues from stalling the overall fetcher throughput. However it usually has the impact of increasing the latency/timeliness of URLs actually being fetched if they are essentially dropped because of low throughput. This means that they are shelved until a future fetch operation. The specific configuration settings is
A more thorough understanding of Fetcher configuration relating to (slow) throughput requires an understanding of the following configuration settings as well
POSSIBLE IMPROVEMENT: It would be advantageous to understand which URLs from which hosts in the queue(s) were resulting in slow throughput. This would facilitate investigation into why this was happening. | ||||||||||||||||||||||
FetcherStatus | hitByTimeLimit | A total count of the URLs dropped across all fetch queues due to the fetcher execution time limit being exceeded. | This metric is valuable for quantifying the number of URLs which have been effectively timebombed e.g. shelved for future fetching due to overall fetcher runtime exceeding a predefined timeout. Although by default the Fetcher never times out e.g. the configuration is set to -1,if a timeout is preferred then the following configuration property can be edited.
POSSIBLE IMPROVEMENT: It could be useful to record the fact that a URL was staged due to it being hit by the timeout limit. This could possibly be stored in the CrawlDatum metadata. Also see
| ||||||||||||||||||||||
FetcherThread | FetcherStatus | AboveExceptionThresholdInQueue | The total count of URLs purged across all fetch queues as a result of the maximum number of protocol-level exceptions (e.g. timeouts) per queue being exceeded. | This metric is useful for quantifying the number of URLs shelved for future fetching due to anomalies occurring during fetcher execution exceeding a predefined ceiling. Although by default the Fetcher never enforces this behaviour e.g. the configuration is set to -1, if this is changed then this total count will become a useful metric to track. Further information on the configuration parameter can be seen below
POSSIBLE IMPROVEMENT: It could be useful to record the fact that a URL was staged due to it being hit by the exception limit. Additionally, it could be useful to write metadata to all URLs which contributed towards the limit being met. This could possibly be stored in the CrawlDatum metadata. | |||||||||||||||||||||
FetcherStatus | FetchItem.notCreated.redirect | A total count of URLs across all fetch queues for which following redirect(s) resulted in no result. | Essentially, each FetcherThread attempts to understand and eliminate all redirect options e.g. duplicate redirect URL, before giving up on a redirect URL entirely. In the case of a redirect URL for which no logical fetch outcome can be produced e.g. that the FetchItem is null, redirecting is simply deactivated as it is impossible to continue. | ||||||||||||||||||||||
FetcherStatus | outlinks_detected | A total count of detected outlinks for all fetch items (URLs) across all fetch queues. | This metric can be used to estimate the number in URLs to be fetched in the next fetch phase. If for example, resources were being allocated within the client at configuration/compile time (rather than dynamically at runtime) this could be used to inform resource reservations, utilization and data partitioning logic. | FetcherStatus | outlinks_detected | ||||||||||||||||||||
FetcherStatus | outlinks_following | ||||||||||||||||||||||||
FetcherStatus | ProtocolStatus.getName() | ||||||||||||||||||||||||
FetcherStatus | redirect_count_exceeded | ||||||||||||||||||||||||
FetcherStatus | redirect_deduplicated | ||||||||||||||||||||||||
FetcherStatus | robots_denied | ||||||||||||||||||||||||
FetcherStatus | robots_denied_maxcrawldelay | ||||||||||||||||||||||||
ParserStatus | ParseStatus.majorCodes[p.getData().getStatus().getMajorCode()] | ||||||||||||||||||||||||
Generator | Generator | EXPR_REJECTED | |||||||||||||||||||||||
Generator | HOSTS_AFFECTED_PER_HOST_OVERFLOW | ||||||||||||||||||||||||
Generator | INTERVAL_REJECTED | ||||||||||||||||||||||||
Generator | MALFORMED_URL | ||||||||||||||||||||||||
Generator | SCHEDULE_REJECTED | ||||||||||||||||||||||||
Generator | SCORE_TOO_LOW | ||||||||||||||||||||||||
Generator | STATUS_REJECTED | ||||||||||||||||||||||||
Generator | URLS_SKIPPED_PER_HOST_OVERFLOW | ||||||||||||||||||||||||
IndexerMapReduce | IndexerStatus | deleted (duplicates) | |||||||||||||||||||||||
IndexerStatus | deleted (IndexingFilter) | ||||||||||||||||||||||||
IndexerStatus | deleted (gone) | ||||||||||||||||||||||||
IndexerStatus | deleted (redirects) | ||||||||||||||||||||||||
IndexerStatus | deleted (robots=noindex) | ||||||||||||||||||||||||
IndexerStatus | errors (IndexingFilter) | ||||||||||||||||||||||||
IndexerStatus | errors (ScoringFilter) | ||||||||||||||||||||||||
IndexerStatus | indexed (add/update) | ||||||||||||||||||||||||
IndexerStatus | skipped (IndexingFilter) | ||||||||||||||||||||||||
IndexerStatus | skipped (not modified) | ||||||||||||||||||||||||
Injector | injector | urls_filtered | |||||||||||||||||||||||
injector | urls_injected | ||||||||||||||||||||||||
injector | urls_merged | ||||||||||||||||||||||||
injector | urls_purged_404 | ||||||||||||||||||||||||
injector | urls_purged_filter | ||||||||||||||||||||||||
ParseSegment | ParserStatus | ParseStatus.majorCodes[parseStatus.getMajorCode()] | |||||||||||||||||||||||
QueueFeeder | FetcherStatus | filtered | |||||||||||||||||||||||
(also QueueFeeder) | FetcherStatus | AboveExceptionThresholdInQueue | |||||||||||||||||||||||
ResolverThread | UpdateHostDb | checked_hosts | |||||||||||||||||||||||
UpdateHostDb | existing_known_host | ||||||||||||||||||||||||
UpdateHostDb | existing_unknown_host | ||||||||||||||||||||||||
UpdateHostDb | new_known_host | ||||||||||||||||||||||||
UpdateHostDb | new_unknown_host | ||||||||||||||||||||||||
UpdateHostDb | purged_unknown_host | ||||||||||||||||||||||||
UpdateHostDb | rediscovered_host | ||||||||||||||||||||||||
UpdateHostDb | Long.toString(datum.numFailures()) + "_times_failed" | ||||||||||||||||||||||||
SitemapProcessor | Sitemap | existing_sitemap_entries | |||||||||||||||||||||||
Sitemap | failed_fetches | ||||||||||||||||||||||||
Sitemap | filtered_records | ||||||||||||||||||||||||
Sitemap | filtered_sitemaps_from_hostname | ||||||||||||||||||||||||
Sitemap | new_sitemap_entries | ||||||||||||||||||||||||
Sitemap | sitemaps_from_hostname | ||||||||||||||||||||||||
Sitemap | sitemap_seeds | ||||||||||||||||||||||||
UpdateHostDbMapper | UpdateHostDb | filtered_records | |||||||||||||||||||||||
UpdateHostDbReducer | UpdateHostDb | total_hosts | |||||||||||||||||||||||
(also UpdateHostDbReducer) | UpdateHostDb | skipped_not_eligible | |||||||||||||||||||||||
WebGraph | WebGraph.outlinks | added links | |||||||||||||||||||||||
(also WebGraph) | WebGraph.outlinks | removed links | |||||||||||||||||||||||
WARCExporter | WARCExporter | exception | |||||||||||||||||||||||
WARCExporter | invalid URI | ||||||||||||||||||||||||
WARCExporter | missing content | ||||||||||||||||||||||||
WARCExporter | missing metadata | ||||||||||||||||||||||||
WARCExporter | omitted empty response | ||||||||||||||||||||||||
WARCExporter | records generated |
...