[TOC]

org.apache.nutch.fetcher

Class OldFetcher

  • java.lang.Object
    • org.apache.hadoop.conf.Configured
    • org.apache.nutch.fetcher.OldFetcher
    • All Implemented Interfaces:
    • org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.MapRunnable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>, org.apache.hadoop.util.Tool 
public class OldFetcher
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable>

The fetcher. Most of the work is done by plugins.

Nested Class Summary

Nested Classes Modifier and Type Class and Description static class OldFetcher.InputFormat

Field Summary

Fields Modifier and Type Field and Description static String CONTENT_REDIR static org.slf4j.Logger LOG static int PERM_REFRESH_TIME static String PROTOCOL_REDIR

Constructor Summary

Constructors Constructor and Description OldFetcher() OldFetcher(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods Modifier and Type Method and Description void close() void configure(org.apache.hadoop.mapred.JobConf job) void fetch(org.apache.hadoop.fs.Path segment, int threads) static boolean isParsing(org.apache.hadoop.conf.Configuration conf) static boolean isStoringContent(org.apache.hadoop.conf.Configuration conf) static void main(String[] args) Run the fetcher. void run(org.apache.hadoop.mapred.RecordReader,org.apache.hadoop.io.Writable> input, org.apache.hadoop.mapred.OutputCollector output, org.apache.hadoop.mapred.Reporter reporter) int run(String[] args) 

-

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf 

-

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait 

-

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

-

LOG

public static final org.slf4j.Logger LOG
-

PERM_REFRESH_TIME

public static final int PERM_REFRESH_TIME
  - See Also:
  - [Constant Field Values](../../../../constant-values.html#org.apache.nutch.fetcher.OldFetcher.PERM_REFRESH_TIME)       
-

CONTENT_REDIR

public static final String CONTENT_REDIR
  - See Also:
  - [Constant Field Values](../../../../constant-values.html#org.apache.nutch.fetcher.OldFetcher.CONTENT_REDIR)       
-

PROTOCOL_REDIR

public static final String PROTOCOL_REDIR
  - See Also:
  - [Constant Field Values](../../../../constant-values.html#org.apache.nutch.fetcher.OldFetcher.PROTOCOL_REDIR)

Constructor Detail

-

OldFetcher

public OldFetcher()
-

OldFetcher

public OldFetcher(org.apache.hadoop.conf.Configuration conf)

Method Detail

-

configure

public void configure(org.apache.hadoop.mapred.JobConf job)
  - Specified by: 
  - <code>configure</code> in interface <code>org.apache.hadoop.mapred.JobConfigurable</code>        
-

close

public void close()
-

isParsing

public static boolean isParsing(org.apache.hadoop.conf.Configuration conf)
-

isStoringContent

public static boolean isStoringContent(org.apache.hadoop.conf.Configuration conf)
-

run

public void run(org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.WritableComparable<?>,org.apache.hadoop.io.Writable> input,
       org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,NutchWritable> output,
       org.apache.hadoop.mapred.Reporter reporter)
         throws IOException
  - Specified by: 
  - <code>run</code> in interface <code>org.apache.hadoop.mapred.MapRunnable<org.apache.hadoop.io.writablecomparable<?>,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Text,NutchWritable&gt;</org.apache.hadoop.io.writablecomparable<?></code> 
  - Throws: 
  - <code>IOException</code>       
-

fetch

public void fetch(org.apache.hadoop.fs.Path segment,
         int threads)
           throws IOException
  - Throws: 
  - <code>IOException</code>       
-

main

public static void main(String[] args)
                 throws Exception

Run the fetcher.

  - Throws: 
  - <code>Exception</code>       
-

run

public int run(String[] args)
        throws Exception
  - Specified by: 
  - <code>run</code> in interface <code>org.apache.hadoop.util.Tool</code> 
  - Throws: 
  - <code>Exception</code>

Copyright © 2014 The Apache Software Foundation