从RocksDB 6.0开始弃用和删除

RocksDB CompactionFilter提供了一种基于后台自定义逻辑删除/过滤过期键/值对的方法。 现在我们在它上面实现了一个扩展,允许用户在Lua中实现自定义的CompactionFilter。这个特性在RocksDB 5.0中可用。

好处

  1. Lua中开发CompactionFilter有以下好处:
  2. 动态更改: 可以在不关闭RocksDB实例的情况下动态更新Lua压缩过滤器。
  3. 单个二进制单元: 由于可以动态地更新Lua压缩过滤器,因此不再需要按照C++压缩过滤器的要求重新构建二进制。
  4. 这对于构建在RocksDB之上的服务来说是一个巨大的好处,RocksDB为客户维护定制的CompactionFilters
  5. 更安全: CompactionFilter中的错误将不再导致核心转储。Lua 引擎捕获所有异常。

如何使用?

  1. 使用RocksLuaCompactionFilter很简单。你需要做的就是以下步骤:
  2. LUA_PATH设置为Lua的根目录,构建RocksDB
  3. lua脚本配置RocksLuaCompactionFilterOptions。(更多细节将在下一节描述)
  4. 使用步骤1中的RocksLuaCompactionFilterOptions构建了一个 RocksLuaCompactionFilterFactory
  5. 将步骤2中的RocksLuaCompactionFilterFactory传递到ColumnFamilyOptions::compaction_filter_factory

例子

  • 这里面有一个简单的RocksLuaCompactionFilter,过滤掉任何键的初始值小于r:

    lua::RocksLuaCompactionFilterOptions lua_opt; // removes all keys whose initial is less than ‘r’ lua_opt.lua_script =

    1. "function Filter(level, key, existing_value)\n"
    2. " if key:sub(1,1) < 'r' then\n"
    3. " return true, false, \"\"\n"
    4. " end\n"
    5. " return false, false, \"\"\n"
    6. "end\n"
    7. "\n"
    8. "function FilterMergeOperand(level, key, operand)\n"
    9. " return false\n"
    10. "end\n"
    11. "function Name()\n"
    12. " return \"KeepsAll\"\n"
    13. "end\n";

    // specify error log. auto* rocks_logger = new facebook::rocks::RocksLogger(

    1. "RocksLuaTest",
    2. true, // print error message in GLOG
    3. true, // print error message to scribe, available in LogView `RocksDB ERROR`
    4. nullptr);

    // Create RocksLuaCompactionFilter with the above lua_opt and pass it to Options rocksdb::Options options; options.compaction_filter_factory =

    1. std::make_shared<rocksdb::lua::RocksLuaCompactionFilterFactory>(lua_opt);

    … // open FbRocksDB with the above options rocksdb::DB* db; auto status = openRocksDB(options, “RocksDBWithLua”, &db);

配置 RocksLuaCompactionFilterOptions

  1. 在这里,我们将更详细地介绍如何配置RocksLuaCompactionFilterOptions
  2. RocksLuaCompactionFilterOptions的定义可以在include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h中找到。

配置Lua脚本(RocksLuaCompactionFilter::Script)

第一个也是最重要的参数是RocksLuaCompactionFilterOptions::script,您的Lua压缩过滤器将在这里实现。Lua脚本必须实现所需的函数,即Name()和Filter()。

  1. // The lua script in string that implements all necessary CompactionFilter
  2. // virtual functions. The specified lua_script must implement the following
  3. // functions, which are Name and Filter, as described below.
  4. //
  5. // 0. The Name function simply returns a string representing the name of
  6. // the lua script. If there's any erorr in the Name function, an
  7. // empty string will be used.
  8. // --- Example
  9. // function Name()
  10. // return "DefaultLuaCompactionFilter"
  11. // end
  12. //
  13. //
  14. // 1. The script must contains a function called Filter, which implements
  15. // CompactionFilter::Filter() , takes three input arguments, and returns
  16. // three values as the following API:
  17. //
  18. // function Filter(level, key, existing_value)
  19. // ...
  20. // return is_filtered, is_changed, new_value
  21. // end
  22. //
  23. // Note that if ignore_value is set to true, then Filter should implement
  24. // the following API:
  25. //
  26. // function Filter(level, key)
  27. // ...
  28. // return is_filtered
  29. // end
  30. //
  31. // If there're any error in the Filter() function, then it will keep
  32. // the input key / value pair.
  33. //
  34. // -- Input
  35. // The function must take three arguments (integer, string, string),
  36. // which map to "level", "key", and "existing_value" passed from
  37. // RocksDB.
  38. //
  39. // -- Output
  40. // The function must return three values (boolean, boolean, string).
  41. // - is_filtered: if the first return value is true, then it indicates
  42. // the input key / value pair should be filtered.
  43. // - is_changed: if the second return value is true, then it indicates
  44. // the existing_value needs to be changed, and the resulting value
  45. // is stored in the third return value.
  46. // - new_value: if the second return value is true, then this third
  47. // return value stores the new value of the input key / value pair.
  48. //
  49. // -- Examples
  50. // -- a filter that keeps all key-value pairs
  51. // function Filter(level, key, existing_value)
  52. // return false, false, ""
  53. // end
  54. //
  55. // -- a filter that keeps all keys and change their values to "Rocks"
  56. // function Filter(level, key, existing_value)
  57. // return false, true, "Rocks"
  58. // end
  59. std::string lua_script;

一个不使用value(RocksLuaCompactionFilter::ignore_value)的优化,以防您的CompactionFilter从不使用value来决定是保留还是丢弃一个键/值对,然后设置RocksLuaCompactionFilterOptions::ignore_value=true并实现简化的Filter() API。 我们的结果表明,这种优化可以节省高达40%的CPU开销引入LuaCompactionFilter:

  1. // If set to true, then existing_value will not be passed to the Filter
  2. // function, and the Filter function only needs to return a single boolean
  3. // flag indicating whether to filter out this key or not.
  4. //
  5. // function Filter(level, key)
  6. // ...
  7. // return is_filtered
  8. // end
  9. bool ignore_value = false;

简化的Filter() API只接受两个输入参数,并且只返回一个布尔标志,指示是保留还是放弃输入键。

Error Log 配置 (RocksLuaCompactionFilterOptions::error_log)

  1. * RocksLuaCompactionFilter 遇到热河错误时,它将作为no-op执行,并且对于任何导致错误的键/值对总是返回false。开发人员可以配置RocksLuaCompactionFilterOptions::error_log 来记录任何Lua错误:
  2. // When specified a non-null pointer, the first "error_limit_per_filter"
  3. // errors of each CompactionFilter that is lua related will be included
  4. // in this log.
  5. std::shared_ptr<Logger> error_log;

注意,对于每个压缩作业,我们只记录error_log中的前几个Lua错误,以避免生成太多的错误消息,并且每个压缩作业报告的错误数量可以通过error_limit_per_filter配置。默认值是1。

  1. // The number of errors per CompactionFilter will be printed
  2. // to error_log.
  3. int error_limit_per_filter = 1;

动态更新Lua脚本

要在运行RocksDB数据库时更新Lua脚本,只需调用RocksLuaCompactionFilterFactory的SetScript() API:

  1. // Change the Lua script so that the next compaction after this
  2. // function call will use the new Lua script.
  3. void SetScript(const std::string& new_script);