HDFS Short-Circuit Local Reads

2017-12-15 10:14:40
Background

In HDFS, reads normally go through the DataNode. Thus, when the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a TCP socket. So-called “short-circuit” reads bypass the DataNode, allowing the client to read the file directly. Obviously, this is only possible in cases where the client is co-located with the data. Short-circuit reads provide a substantial performance boost to many applications

Setup

To configure short-circuit local reads, you will need to enable libhadoop.so. See Native Libraries for details on enabling this library. if you don’t compile yourself, may be use complile binary version ,see here
Short-circuit reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate. You will need to set a path to this socket. The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the HDFS user or root to create this path. For this reason, paths under /var/run or /var/lib are often used.

The client and the DataNode exchange information via a shared memory segment on /dev/shm.

Short-circuit local reads need to be configured on both the DataNode and the client.

Java can not use Unix Domain Socket directly，so you need install Hadoop native package libhadoop.so。if you use Pivotal HD，CDH and so on, native package will be install at you install hadoop package. you can use command to check native package like this

1
2
3
4
5
6
7
8
9
10
11
$ hadoop checknative
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:   true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4:    true revision:99
bzip2:  true /lib64/libbz2.so.1


#如果要排查问题,可以更改日志级别，进行相应排查
export HADOOP_ROOT_LOGGER=DEBUG,console
$ hadoop checknative -a

#打开short-circuit local reads 功能

dfs.client.read.shortcircuit: false

#可选。该参数是一个指向UNIX域套接字的路径，用于DataNode和本地HDFS客户端通信。如果在该路径中出现了字符串”_PORT”，会被替换成DataNode的TCP端口。

dfs.domain.socket.path:

#设置了该参数，short-circuit local reads功能将跳过checksums校验。通常不推荐这么做，但是该参数对于特殊场合可能有用。如果你在HDFS之外自己做checksum校验，那么就该考虑设置该参数。

dfs.client.read.shortcircuit.skip.checksum: false

#DFSClient维护着一个用于保存最近已打开的文件描述符的缓存。该参数控制着此缓存的容量。增大该缓存的容量就可以使用更多文件描述符，但是，在涉及大量seek操作的负载上可能带来更好的性能

dfs.client.read.shortcircuit.streams.cache.size: 256

#该参数控制着文件描述符因为长期不活跃而被关闭之前需要在客户端缓存上下文中驻留的最小时间

dfs.client.read.shortcircuit.streams.cache.expiry.ms: 300000

Example Configuration
1
2
3
4
5
6
7
8
9
10
<configuration>
  <property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.domain.socket.path</name>
    <value>/var/lib/hadoop-hdfs/dn_socket</value>
  </property>
</configuration>
Legacy HDFS Short-Circuit Local Reads

Legacy implementation of short-circuit local reads on which the clients directly open the HDFS block files is still available for platforms other than the Linux. Setting the value of dfs.client.use.legacy.blockreader.local in addition to dfs.client.read.shortcircuit to true enables this feature.

You also need to set the value of dfs.datanode.data.dir.perm to 750 instead of the default 700 and chmod/chown the directory tree under dfs.datanode.data.dir as readable to the client and the DataNode. You must take caution because this means that the client can read all of the block files bypassing HDFS permission.

Because Legacy short-circuit local reads is insecure, access to this feature is limited to the users listed in the value of dfs.block.local-path-access.user.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<configuration>
  <property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.client.use.legacy.blockreader.local</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir.perm</name>
    <value>750</value>
  </property>
  <property>
    <name>dfs.block.local-path-access.user</name>
    <value>foo,bar</value>
  </property>
</configuration>
碰到的问题(centos7 hadoop2.9)

1.openssl问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@namenode ~]# export HADOOP_OPTS=-Djava.library.path=/usr/local/hadoop-2.9.0/lib/native
[root@namenode ~]# hadoop checknative -a
17/12/25 15:46:06 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
17/12/25 15:46:06 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
17/12/25 15:46:07 DEBUG util.Shell: setsid exited with exit code 0
17/12/25 15:46:07 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
17/12/25 15:46:07 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
17/12/25 15:46:07 DEBUG crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
        at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
        at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:87)
        at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:101)
Native library checking:
hadoop:  true /usr/local/hadoop-2.9.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /lib64/libsnappy.so.1
zstd  :  false 
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
17/12/25 15:46:07 DEBUG util.ExitUtil: Exiting with status 1: ExitException
1: ExitException
        at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
        at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)
        at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:145)
17/12/25 15:46:07 INFO util.ExitUtil: Exiting with status 1: ExitException

解决方法：

1
ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so

2.zstd问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@namenode ~]# hadoop checknative -a
17/12/25 17:24:43 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
17/12/25 17:24:43 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
17/12/25 17:24:43 DEBUG util.Shell: setsid exited with exit code 0
17/12/25 17:24:43 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
17/12/25 17:24:43 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/local/hadoop-2.9.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /lib64/libsnappy.so.1
zstd  :  false 
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
17/12/25 17:24:43 DEBUG util.ExitUtil: Exiting with status 1: ExitException
1: ExitException
        at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:304)
        at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:292)
        at org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:145)
17/12/25 17:24:43 INFO util.ExitUtil: Exiting with status 1: ExitException

解决方法：无,做了以下尝试，无结果

1
2
yum install epel-release -y
yum install libzstd -y

后查看源代码，在源码包中查看BUILDING.txt文件，发现编译环境没有libzstd，猜测可能编译时没有安装，但代码里有检测，所以有提示

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
public class NativeCodeLoader {

  private static final Logger LOG =
      LoggerFactory.getLogger(NativeCodeLoader.class);
  
  private static boolean nativeCodeLoaded = false;
  
  static {
    // Try to load native hadoop library and set fallback flag appropriately
    if(LOG.isDebugEnabled()) {
      LOG.debug("Trying to load the custom-built native-hadoop library...");
    }
    try {
      System.loadLibrary("hadoop");
      LOG.debug("Loaded the native-hadoop library");
      nativeCodeLoaded = true;
    } catch (Throwable t) {
      // Ignore failure to load
      if(LOG.isDebugEnabled()) {
        LOG.debug("Failed to load native-hadoop with error: " + t);
        LOG.debug("java.library.path=" +
            System.getProperty("java.library.path"));
      }
    }
    
    if (!nativeCodeLoaded) {
      LOG.warn("Unable to load native-hadoop library for your platform... " +
               "using builtin-java classes where applicable");
    }
  }

  /**
   * Check if native-hadoop code is loaded for this platform.
   * 
   * @return <code>true</code> if native-hadoop is loaded, 
   *         else <code>false</code>
   */
  public static boolean isNativeCodeLoaded() {
    return nativeCodeLoaded;
  }

  /**
   * Returns true only if this build was compiled with support for snappy.
   */
  public static native boolean buildSupportsSnappy();
  
  /**
   * Returns true only if this build was compiled with support for ZStandard.
   */
  public static native boolean buildSupportsZstd();

  /**
   * Returns true only if this build was compiled with support for openssl.
   */
  public static native boolean buildSupportsOpenssl();

  public static native String getLibraryName();

  /**
   * Return if native hadoop libraries, if present, can be used for this job.
   * @param conf configuration
   * 
   * @return <code>true</code> if native hadoop libraries, if present, can be 
   *         used for this job; <code>false</code> otherwise.
   */
  public boolean getLoadNativeLibraries(Configuration conf) {
    return conf.getBoolean(CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY, 
                           CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_DEFAULT);
  }
  
  /**
   * Set if native hadoop libraries, if present, can be used for this job.
   * 
   * @param conf configuration
   * @param loadNativeLibraries can native hadoop libraries be loaded
   */
  public void setLoadNativeLibraries(Configuration conf, 
                                     boolean loadNativeLibraries) {
    conf.setBoolean(CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY,
                    loadNativeLibraries);
  }

}

ref
Short-Circuit Local Reads
详解HDFS Short Circuit Local Reads
Unable to load native-hadoop library for your platform

您的鼓励是我写作最大的动力

俗话说，投资效率是最好的投资。 如果您感觉我的文章质量不错，读后收获很大，预计能为您提高 10% 的工作效率，不妨小额捐助我一下，让我有动力继续写出更多好文章。