maven 安装

在编译过程中出现一下错误：

  1. [ERROR] Plugin org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-enforcer-plugin:jar:3.0.0-M1: Could not transfer artifact org.apache.maven.plugins:maven-enforcer-plugin:pom:3.0.0-M1 from/to central (https://repo.maven.apache.org/maven2): Received fatal alert: protocol_version -> [Help 1]
  2. [ERROR] 
  3. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  4. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  5. [ERROR] 
  6. [ERROR] For more information about the errors and possible solutions, please read the following articles:
  7. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
  8. [root@cdh2 YCSB-0.13.0]# ls

解决方案；

  1. 将maven文件里的源修改为阿里云源
  3. <mirror>
  4.       <id>alimaven</id>
  5.       <name>aliyun maven</name>
  6.       <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
  7.       <mirrorOf>central</mirrorOf>       
  8. </mirror>
  1. YCSB使用说明
    [root@cdh2 ycsb]# cat hbase098-binding/README.md

    HBase (0.98.x) Driver for YCSB

This driver is a binding for the YCSB facilities to operate against a HBase 0.98.x Server cluster.
To run against an HBase 0.94.x cluster, use the hbase094 binding.
To run against an HBase >= 1.0 cluster, use the hbase10 binding.

Quickstart

1. Start a HBase Server

You need to start a single node or a cluster to point the client at. Please see Apache HBase Reference Guide for more details and instructions.

2. Set up YCSB

You need to clone the repository and compile everything.

  1. git clone git://github.com/brianfrankcooper/YCSB.git
  2. cd YCSB
  3. mvn clean package

3. Create a HBase table for testing

For best results, use the pre-splitting strategy recommended in HBASE-4163:

  1. hbase(main):001:0> n_splits = 200 # HBase recommends (10 * number of regionservers)
  2. hbase(main):002:0> create 'usertable', 'family', {SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-1000)/n_splits}"}}

Failing to do so will cause all writes to initially target a single region server.

4. Run the Workload

Before you can actually run the workload, you need to “load” the data first.

You should specify a HBase config directory(or any other directory containing your hbase-site.xml) and a table name and a column family(-cp is used to set java classpath and -p is used to set various properties).

  1. bin/ycsb load hbase -P workloads/workloada -cp /HBASE-HOME-DIR/conf -p table=usertable -p columnfamily=family

Then, you can run the workload:

  1. bin/ycsb run hbase -P workloads/workloada -cp /HBASE-HOME-DIR/conf -p table=usertable -p columnfamily=family

Please see the general instructions in the doc folder if you are not sure how it all works. You can apply additional properties (as seen in the next section) like this:

  1. bin/ycsb run hbase -P workloads/workloada -cp /HBASE-HOME-DIR/conf -p table=usertable -p columnfamily=family -p clientbuffering=true

Configuration Options

Following options can be configurable using -p.

  • columnfamily: The HBase column family to target.
  • debug : If true, debugging logs are activated. The default is false.
  • hbase.usepagefilter : If true, HBase
    PageFilters
    are used to limit the number of records consumed in a scan operation. The default is true.
  • principal: If testing need to be done against a secure HBase cluster using Kerberos Keytab,
    this property can be used to pass the principal in the keytab file.
  • keytab: The Kerberos keytab file name and location can be passed through this property.
  • writebuffersize: The maximum amount, in bytes, of data to buffer on the client side before a flush is forced. The default is 12MB.

Additional HBase settings should be provided in the hbase-site.xml file located in your /HBASE-HOME-DIR/conf directory. Typically this will be /etc/hbase/conf.[root@cdh2 ycsb]#

  1. HBase中创建测试表
  1. hbase(main):001:0> n_splits = 200 # HBase recommends (10 * number of regionservers)
  2. hbase(main):002:0> create 'usertable', 'family', {SPLITS => (1..n_splits).map {|i| "user#{1000+i*(9999-1000)/n_splits}"}}
  5. create 'usertable', {NAME=>'f1', VERSIONS=>'1',COMPRESSION=>'LZO'}

加载

  1. bin/ycsb load hbase -P workloads/workloada -cp /HBASE-HOME-DIR/conf -p table=usertable -p columnfamily=family
  3. 例如：
  4. bin/ycsb load hbase10 -P workloads/workloada -cp /etc/hbase/conf.cloudera.hbase/ -p table=usertable -p columnfamily=family
  7. 因为ycsb是精简版的hbase10-binding目录下没有conf目录，需要制定本地hbase的配置文件所在目录，即/etc/hbase/conf.cloudera.hbase
  11. 注意：
  12. 由于本人使用cdh安装的hbase,配置文件所在的目录在/etc/hbase/conf.cloudera.hbase目录，与bin安装的方式有所区别

打印log

  1. Loading workload...
  2. Starting test.
  3. 19/04/25 18:29:21 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x66886381 connecting to ZooKeeper ensemble=cdh3:2181
  4. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
  5. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:host.name=cdh2
  6. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_80
  7. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
  8. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_80/jre
  9. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/etc/hbase/conf.cloudera.hbase/:/soft/ycsb-0.13.0/hbase10-binding/conf:/soft/ycsb-0.13.0/conf:/soft/ycsb-0.13.0/lib/core-0.13.0.jar:/soft/ycsb-0.13.0/lib/htrace-core4-4.1.0-incubating.jar:/soft/ycsb-0.13.0/lib/jackson-mapper-asl-1.9.4.jar:/soft/ycsb-0.13.0/lib/jackson-core-asl-1.9.4.jar:/soft/ycsb-0.13.0/lib/HdrHistogram-2.1.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/activation-1.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/apacheds-i18n-2.0.0-M15.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/api-asn1-api-1.0.0-M20.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/api-util-1.0.0-M20.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/avro-1.7.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-beanutils-1.7.0.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-beanutils-core-1.8.0.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-cli-1.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-codec-1.9.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-collections-3.2.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-compress-1.4.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-configuration-1.6.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-digester-1.8.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-el-1.0.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-httpclient-3.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-io-2.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-lang-2.6.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-logging-1.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-math3-3.1.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/commons-net-3.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/findbugs-annotations-1.3.9-1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/guava-12.0.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-annotations-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-auth-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-common-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-mapreduce-client-core-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-yarn-api-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hadoop-yarn-common-2.5.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hbase-annotations-1.0.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hbase-client-1.0.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hbase-common-1.0.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hbase-protocol-1.0.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/htrace-core-3.1.0-incubating.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/httpclient-4.2.5.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/httpcore-4.2.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jackson-core-asl-1.9.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jackson-mapper-asl-1.8.8.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jaxb-api-2.2.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jcodings-1.0.8.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jetty-util-6.1.26.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/joni-2.1.2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jsch-0.1.42.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/jsr305-1.3.9.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/log4j-1.2.17.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/netty-3.6.2.Final.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/netty-all-4.0.23.Final.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/paranamer-2.3.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/protobuf-java-2.5.0.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/slf4j-api-1.6.4.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/slf4j-log4j12-1.7.7.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/snappy-java-1.0.4.1.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/stax-api-1.0-2.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/xmlenc-0.52.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/xz-1.0.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/zookeeper-3.4.6.jar:/soft/ycsb-0.13.0/hbase10-binding/lib/hbase10-binding-0.13.0.jar
  10. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
  11. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
  12. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
  13. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
  14. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
  15. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
  16. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:user.name=root
  17. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
  18. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Client environment:user.dir=/soft/ycsb-0.13.0
  19. 19/04/25 18:29:21 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cdh3:2181 sessionTimeout=60000 watcher=hconnection-0x668863810x0, quorum=cdh3:2181, baseZNode=/hbase
  20. 19/04/25 18:29:21 INFO zookeeper.ClientCnxn: Opening socket connection to server cdh3/192.168.0.142:2181. Will not attempt to authenticate using SASL (unknown error)
  21. 19/04/25 18:29:21 INFO zookeeper.ClientCnxn: Socket connection established to cdh3/192.168.0.142:2181, initiating session
  22. 19/04/25 18:29:21 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh3/192.168.0.142:2181, sessionid = 0x16a53e9b80e000a, negotiated timeout = 60000
  23. 19/04/25 18:29:21 WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://cdh1:8020/hbase/lib, ignored
  24. java.io.IOException: No FileSystem for scheme: hdfs
  25.     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2579)
  26.     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586)
  27.     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
  28.     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
  29.     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
  30.     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
  31.     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
  32.     at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:104)
  33.     at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:232)
  34.     at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
  35.     at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
  36.     at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
  37.     at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
  38.     at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
  39.     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  40.     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  41.     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  42.     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  43.     at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
  44.     at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
  45.     at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
  46.     at com.yahoo.ycsb.db.HBaseClient10.init(HBaseClient10.java:152)
  47.     at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)
  48.     at com.yahoo.ycsb.ClientThread.run(Client.java:423)
  49.     at java.lang.Thread.run(Thread.java:745)
  50. DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
  51. 19/04/25 18:29:34 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
  52. 19/04/25 18:29:34 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16a53e9b80e000a
  53. 19/04/25 18:29:34 INFO zookeeper.ZooKeeper: Session: 0x16a53e9b80e000a closed
  54. 19/04/25 18:29:34 INFO zookeeper.ClientCnxn: EventThread shut down
  55. [OVERALL], RunTime(ms), 14862  // 数据加载所用时间：14862 14.862秒
  56. [OVERALL], Throughput(ops/sec), 67.28569506122999 // 加载操作的吞吐量，平均并发量每条67.28条
  57. [TOTAL_GCS_MarkSweepCompact], Count, 0
  58. [TOTAL_GC_TIME_MarkSweepCompact], Time(ms), 0
  59. [TOTAL_GC_TIME_%_MarkSweepCompact], Time(%), 0.0
  60. [TOTAL_GCS_Copy], Count, 8
  61. [TOTAL_GC_TIME_Copy], Time(ms), 229
  62. [TOTAL_GC_TIME_%_Copy], Time(%), 1.5408424169021666
  63. [TOTAL_GCs], Count, 8
  64. [TOTAL_GC_TIME], Time(ms), 229
  65. [TOTAL_GC_TIME_%], Time(%), 1.5408424169021666
  66. [CLEANUP], Operations, 2  执行cleanup的操作总数，2
  67. [CLEANUP], AverageLatency(us), 61237.0 // 平均响应时间61.237ms
  68. [CLEANUP], MinLatency(us), 10  // 最小响应时间 0.01
  69. [CLEANUP], MaxLatency(us), 122495 // 最大响应时间 122.495ms
  70. [CLEANUP], 95thPercentileLatency(us), 122495 // 95%的cleanup操作延时在122.495ms以内
  71. [CLEANUP], 99thPercentileLatency(us), 122495 // 99%的cleanup操作延时在122.495ms以内
  72. [INSERT], Operations, 1000 // 执行insert操作总数，1000
  73. [INSERT], AverageLatency(us), 12081.785 // 每次insert操作的平均延时12.081ms
  74. [INSERT], MinLatency(us), 3234 // 所有insert操作最小延时，3.234ms以内
  75. [INSERT], MaxLatency(us), 440319 // 所有insert操作最大延时，440.319ms
  76. [INSERT], 95thPercentileLatency(us), 34751 95%的insert操作延时在34.751ms以内
  77. [INSERT], 99thPercentileLatency(us), 78399 // 99%的insert操作延时在78.399ms 以内
  78. [INSERT], Return=OK, 1000  成功返回数 1000

运行

  1. bin/ycsb run hbase -P workloads/workloada -cp /HBASE-HOME-DIR/conf -p table=usertable -p columnfamily=family -p clientbuffering=true
  3. 例如：
  4. bin/ycsb run hbase10 -P workloads/workloada -cp /etc/hbase/conf.cloudera.hbase/ -p table=usertable -p columnfamily=family

打印log

  1. [OVERALL], RunTime(ms), 5637 // 数据加载所用时间：5.637
  2. [OVERALL], Throughput(ops/sec), 177.39932588256164 // 加载操作的吞吐量，平均并发量每秒177.39条
  3. [TOTAL_GCS_MarkSweepCompact], Count, 0
  4. [TOTAL_GC_TIME_MarkSweepCompact], Time(ms), 0
  5. [TOTAL_GC_TIME_%_MarkSweepCompact], Time(%), 0.0
  6. [TOTAL_GCS_Copy], Count, 7
  7. [TOTAL_GC_TIME_Copy], Time(ms), 77
  8. [TOTAL_GC_TIME_%_Copy], Time(%), 1.3659748092957247
  9. [TOTAL_GCs], Count, 7
  10. [TOTAL_GC_TIME], Time(ms), 77
  11. [TOTAL_GC_TIME_%], Time(%), 1.3659748092957247
  12. [CLEANUP], Operations, 1 //执行cleanup的操作总数，1
  13. [CLEANUP], AverageLatency(us), 578304.0 // 平均响应时间：57.8304
  14. [CLEANUP], MinLatency(us), 578048 // 最小相应时间0.578ms
  15. [CLEANUP], MaxLatency(us), 578559 //最大响应时间0.578ms
  16. [CLEANUP], 95thPercentileLatency(us), 578559 // 95%的insert操作延时在0.578ms以内
  17. [CLEANUP], 99thPercentileLatency(us), 578559 // 99%的insert操作延时在0.578ms以内
  18. [READ], Operations, 506  // 执行read操作总数，506次
  19. [READ], AverageLatency(us), 4099.938735177866  // 平均响应时间4.099ms
  20. [READ], MinLatency(us), 910  // 最小响应时间0.91ms
  21. [READ], MaxLatency(us), 442623 // 最大响应时间44.2623ms
  22. [READ], 95thPercentileLatency(us), 10479 // 95%的read操作延时在10.479ms
  23. [READ], 99thPercentileLatency(us), 26911 // 99%的insert操作延时在26.11ms以内
  24. [READ], Return=OK, 506 成功返回506
  25. [UPDATE], Operations, 495
  26. [UPDATE], AverageLatency(us), 1019.2525252525253
  27. [UPDATE], MinLatency(us), 31
  28. [UPDATE], MaxLatency(us), 451583
  29. [UPDATE], 95thPercentileLatency(us), 95
  30. [UPDATE], 99thPercentileLatency(us), 127
  31. [UPDATE], Return=OK, 494

测试用例

  1. 目录结构
  2. [root@cdh2 hbase_ycsb_testcase]# ll
  3. 总用量 0
  4. -rw-r--r-- 1 root root 0 4月  25 19:13 TR1003
  5. -rw-r--r-- 1 root root 0 4月  25 19:14 TR1003.report
  6. -rw-r--r-- 1 root root 0 4月  25 19:14 ycsb_load.sh
  7. -rw-r--r-- 1 root root 0 4月  25 19:14 ycsb_run.sh
  8. [root@cdh2 hbase_ycsb_testcase]#

ycsb_load.sh脚本的内容

  1. echo "******Loading test begin******"
  5. # define the path of ycsb
  7. YCSB="/root/ycsb-hbase10-binding-0.10.0"
  11. # define the path of  hbase_site.xml
  13. hbase=$YCSB"/bin"
  17. # define test path
  19. test=$YCSB"/workloads/test/"
  23. # define the file path of workload
  25. workload=$test$1
  29. # define the log file name
  31. report=$test$1".report"
  35. # define ycbs runner
  37. runner=$YCSB"/bin/ycsb"
  41. # define measurement param
  43. raw_file=$test$1".raw"
  45. measurement_param="measurement.raw.output_file="$raw_file
  49. # run test
  51. $runner load hbase10 -cp $hbase -P $workload -s -jvm-args='-Xmx32g' 1>>$report 2>>$report
  55. echo "******Loading test end******"
  1. 脚本内容较为简单，ycsb_run.sh脚本的内容与上述内容几乎一样（最后一句的ycsb运行命令中的load改为run），其中$1变量时用户在运行时所指定的配置文件（workload、TR1003等）。
  3. -jvm-args='-Xmx32g' 这个是用来配置ycsb运行时jvm虚拟机的内存大小的，这里的意思是最多给ycsb clinet进程分配32G的内存
  5. 具体用法：sh ycsb_load.sh TR1003

ycsb配置文件TR1003（作用跟workload文件相同）

  1. # The thread count
  3. threadcount=100
  7. # The number of fields in a record
  9. fieldcount=1
  13. # The size of each field (in bytes)
  15. fieldlength=9216
  19. # Number of Records will be loaded
  21. recordcount=1500000000
  25. # Number of Operations will be handle in run parsh
  27. operationcount=2000000
  29. readallfields=true
  31. #insertorder=hashed
  33. #insertstart=0
  35. #insertcount=500000000
  39. # Control Porption of hbase operation type
  41. readproportion=0
  43. updateproportion=0
  45. scanproportion=1
  47. insertproportion=0
  51. # The following param always be fixed
  53. # The table name
  55. table=usertable
  59. # The colume family
  61. columnfamily=cf
  65. # The workload class
  67. workload=com.yahoo.ycsb.workloads.CoreWorkload
  71. # The measurement type
  73. measurementtype=raw
  77. maxscanlength=1000000
  81. #hbase.usepagefilter=false
  85. #scanlengthdistribution=zipfian
  89. #requestdistribution=latest
  1. 这个配置文件是单独用来load阶段加载数据库时候用的，其中clientbuffer配置项是配置hbase client端的写缓存，配置写缓存有利用减少写hbase操作时的rpc开销，具体作用可以查看hbase测试调优文档，默认是不配的，该配置项默认的值是配置项writebuffersize=1024*1024*12（12M），requestdistribution是指操作需求分布采用的方式，。

Run阶段配置文件TR1003：

  1. # The thread count
  3. threadcount=100
  7. # The number of fields in a record
  9. fieldcount=1
  13. # The size of each field (in bytes)
  15. fieldlength=9216
  19. # Number of Records will be loaded
  21. recordcount=1500000000
  25. # Number of Operations will be handle in run parsh
  27. operationcount=2000000
  29. readallfields=true
  31. #insertorder=hashed
  33. #insertstart=0
  35. #insertcount=500000000
  39. # Control Porption of hbase operation type
  41. readproportion=0
  43. updateproportion=0
  45. scanproportion=1
  47. insertproportion=0
  51. # The following param always be fixed
  53. # The table name
  55. table=usertable
  59. # The colume family
  61. columnfamily=cf
  65. # The workload class
  67. workload=com.yahoo.ycsb.workloads.CoreWorkload
  71. # The measurement type
  73. measurementtype=raw
  77. maxscanlength=1000000
  81. #hbase.usepagefilter=false
  85. #scanlengthdistribution=zipfian
  89. #requestdistribution=latest

说明

  1. 这个配置文件用在run阶段，与ycsb_run.sh配合使用，这里run阶段主要就是读hbase数据库的数据。
  3. scan：
  5. Maxscanlength配置项用来指定scan的条数（在ycsb代码中的实现方式是取1~Maxscanlength中的随机值作为一次scan的条数）
  7. hbase.usepagefilter配置项是scan一次的结果是否采取分页显示，默认是开启的
  9. scanlengthdistribution配置项决定了以何种方式从1~Maxscanlength区间中取值，还是默认的uniform（等概率随机分布）。
  11. 另外值得注意的是在执行scan操作中，ycsb每秒操作数currentops/sec可能会很小：
  13. 2016-08-30 04:21:40:791 60 sec: 1 operations; 0.1 current ops/sec; est completion in 1388 days 21hours SCAN count: 1, average latency(us): 49083737.00
  15. 这个是由于ycsb计算方式的原因，它只计算每次具体的操作数，而不是scan总数，所以0.1可以理解为在10秒钟的时间内执行了1次scan操作，所以这10秒的平均操作数是0.1 current ops/sec。
  17. 对于scan操作我们可以通过带宽来计算一次操作到底扫描了多少条数据。

注意

  1. Load阶段其实就是装载数据即往hbase中插入数据，workload文件中的recordcount就是将要插入的条数，run阶段就是对hbase进行各种操作，operationcount就是操作数，因此load阶段必须要正确执行完成，否则在run阶段会有类似于[READ-FAILED]之类的错误。

ycsb的帮助信息如下：

  1. [root@cdh2 ycsb]# ycsb --help
  2. usage: /soft/ycsb/bin/ycsb command database [options]
  4. Commands:
  5.     load           Execute the load phase
  6.     run            Execute the transaction phase
  7.     shell          Interactive mode
  9. Databases:
  10.     accumulo       https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
  11.     aerospike      https://github.com/brianfrankcooper/YCSB/tree/master/aerospike
  12.     arangodb       https://github.com/brianfrankcooper/YCSB/tree/master/arangodb
  13.     arangodb3      https://github.com/brianfrankcooper/YCSB/tree/master/arangodb3
  14.     asynchbase     https://github.com/brianfrankcooper/YCSB/tree/master/asynchbase
  15.     azuredocumentdb https://github.com/brianfrankcooper/YCSB/tree/master/azuredocumentdb
  16.     azuretablestorage https://github.com/brianfrankcooper/YCSB/tree/master/azuretablestorage
  17.     basic          https://github.com/brianfrankcooper/YCSB/tree/master/basic
  18.     basicts        https://github.com/brianfrankcooper/YCSB/tree/master/basicts
  19.     cassandra-cql  https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
  20.     cassandra2-cql https://github.com/brianfrankcooper/YCSB/tree/master/cassandra2
  21.     cloudspanner   https://github.com/brianfrankcooper/YCSB/tree/master/cloudspanner
  22.     couchbase      https://github.com/brianfrankcooper/YCSB/tree/master/couchbase
  23.     couchbase2     https://github.com/brianfrankcooper/YCSB/tree/master/couchbase2
  24.     dynamodb       https://github.com/brianfrankcooper/YCSB/tree/master/dynamodb
  25.     elasticsearch  https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch
  26.     elasticsearch5 https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch5
  27.     geode          https://github.com/brianfrankcooper/YCSB/tree/master/geode
  28.     googlebigtable https://github.com/brianfrankcooper/YCSB/tree/master/googlebigtable
  29.     googledatastore https://github.com/brianfrankcooper/YCSB/tree/master/googledatastore
  30.     hbase094       https://github.com/brianfrankcooper/YCSB/tree/master/hbase094
  31.     hbase098       https://github.com/brianfrankcooper/YCSB/tree/master/hbase098
  32.     hbase10        https://github.com/brianfrankcooper/YCSB/tree/master/hbase10
  33.     hbase12        https://github.com/brianfrankcooper/YCSB/tree/master/hbase12
  34.     hypertable     https://github.com/brianfrankcooper/YCSB/tree/master/hypertable
  35.     infinispan     https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
  36.     infinispan-cs  https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
  37.     jdbc           https://github.com/brianfrankcooper/YCSB/tree/master/jdbc
  38.     kudu           https://github.com/brianfrankcooper/YCSB/tree/master/kudu
  39.     memcached      https://github.com/brianfrankcooper/YCSB/tree/master/memcached
  40.     mongodb        https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
  41.     mongodb-async  https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
  42.     nosqldb        https://github.com/brianfrankcooper/YCSB/tree/master/nosqldb
  43.     orientdb       https://github.com/brianfrankcooper/YCSB/tree/master/orientdb
  44.     rados          https://github.com/brianfrankcooper/YCSB/tree/master/rados
  45.     redis          https://github.com/brianfrankcooper/YCSB/tree/master/redis
  46.     rest           https://github.com/brianfrankcooper/YCSB/tree/master/rest
  47.     riak           https://github.com/brianfrankcooper/YCSB/tree/master/riak
  48.     s3             https://github.com/brianfrankcooper/YCSB/tree/master/s3
  49.     solr           https://github.com/brianfrankcooper/YCSB/tree/master/solr
  50.     solr6          https://github.com/brianfrankcooper/YCSB/tree/master/solr6
  51.     tarantool      https://github.com/brianfrankcooper/YCSB/tree/master/tarantool
  53. Options:
  54.     -P file        Specify workload file
  55.     -cp path       Additional Java classpath entries
  56.     -jvm-args args Additional arguments to the JVM
  57.     -p key=value   Override workload property
  58.     -s             Print status to stderr
  59.     -target n      Target ops/sec (default: unthrottled)
  60.     -threads n     Number of client threads (default: 1)
  62. Workload Files:
  63.     There are various predefined workloads under workloads/ directory.
  64.     See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
  65.     for the list of workload properties.
  67. positional arguments:
  68.   {load,run,shell}      Command to run.
  69.   {accumulo,aerospike,arangodb,arangodb3,asynchbase,azuredocumentdb,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,dynamodb,elasticsearch,elasticsearch5,geode,googlebigtable,googledatastore,hbase094,hbase098,hbase10,hbase12,hypertable,infinispan,infinispan-cs,jdbc,kudu,memcached,mongodb,mongodb-async,nosqldb,orientdb,rados,redis,rest,riak,s3,solr,solr6,tarantool}
  70.                         Database to test.
  72. optional arguments:
  73.   -h, --help            show this help message and exit
  74.   -cp CLASSPATH         Additional classpath entries, e.g. '-cp
  75.                         /tmp/hbase-1.0.1.1/conf'. Will be prepended to the
  76.                         YCSB classpath.
  77.   -jvm-args JVM_ARGS    Additional arguments to pass to 'java', e.g. '-Xmx4g'