从RocksDB 6.0开始弃用和删除
RocksDB CompactionFilter提供了一种基于后台自定义逻辑删除/过滤过期键/值对的方法。 现在我们在它上面实现了一个扩展,允许用户在Lua中实现自定义的CompactionFilter。这个特性在RocksDB 5.0中可用。
好处
在Lua中开发CompactionFilter有以下好处:动态更改: 可以在不关闭RocksDB实例的情况下动态更新Lua压缩过滤器。单个二进制单元: 由于可以动态地更新Lua压缩过滤器,因此不再需要按照C++压缩过滤器的要求重新构建二进制。这对于构建在RocksDB之上的服务来说是一个巨大的好处,RocksDB为客户维护定制的CompactionFilters。更安全: CompactionFilter中的错误将不再导致核心转储。Lua 引擎捕获所有异常。
如何使用?
使用RocksLuaCompactionFilter很简单。你需要做的就是以下步骤:将LUA_PATH设置为Lua的根目录,构建RocksDB。用lua脚本配置RocksLuaCompactionFilterOptions。(更多细节将在下一节描述)使用步骤1中的RocksLuaCompactionFilterOptions构建了一个 RocksLuaCompactionFilterFactory。将步骤2中的RocksLuaCompactionFilterFactory传递到ColumnFamilyOptions::compaction_filter_factory。
例子
这里面有一个简单的RocksLuaCompactionFilter,过滤掉任何键的初始值小于r:
lua::RocksLuaCompactionFilterOptions lua_opt; // removes all keys whose initial is less than ‘r’ lua_opt.lua_script =
"function Filter(level, key, existing_value)\n"" if key:sub(1,1) < 'r' then\n"" return true, false, \"\"\n"" end\n"" return false, false, \"\"\n""end\n""\n""function FilterMergeOperand(level, key, operand)\n"" return false\n""end\n""function Name()\n"" return \"KeepsAll\"\n""end\n";
// specify error log. auto* rocks_logger = new facebook::rocks::RocksLogger(
"RocksLuaTest",true, // print error message in GLOGtrue, // print error message to scribe, available in LogView `RocksDB ERROR`nullptr);
// Create RocksLuaCompactionFilter with the above lua_opt and pass it to Options rocksdb::Options options; options.compaction_filter_factory =
std::make_shared<rocksdb::lua::RocksLuaCompactionFilterFactory>(lua_opt);
… // open FbRocksDB with the above options rocksdb::DB* db; auto status = openRocksDB(options, “RocksDBWithLua”, &db);
配置 RocksLuaCompactionFilterOptions
在这里,我们将更详细地介绍如何配置RocksLuaCompactionFilterOptions。RocksLuaCompactionFilterOptions的定义可以在include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h中找到。
配置Lua脚本(RocksLuaCompactionFilter::Script)
第一个也是最重要的参数是RocksLuaCompactionFilterOptions::script,您的Lua压缩过滤器将在这里实现。Lua脚本必须实现所需的函数,即Name()和Filter()。
// The lua script in string that implements all necessary CompactionFilter// virtual functions. The specified lua_script must implement the following// functions, which are Name and Filter, as described below.//// 0. The Name function simply returns a string representing the name of// the lua script. If there's any erorr in the Name function, an// empty string will be used.// --- Example// function Name()// return "DefaultLuaCompactionFilter"// end////// 1. The script must contains a function called Filter, which implements// CompactionFilter::Filter() , takes three input arguments, and returns// three values as the following API://// function Filter(level, key, existing_value)// ...// return is_filtered, is_changed, new_value// end//// Note that if ignore_value is set to true, then Filter should implement// the following API://// function Filter(level, key)// ...// return is_filtered// end//// If there're any error in the Filter() function, then it will keep// the input key / value pair.//// -- Input// The function must take three arguments (integer, string, string),// which map to "level", "key", and "existing_value" passed from// RocksDB.//// -- Output// The function must return three values (boolean, boolean, string).// - is_filtered: if the first return value is true, then it indicates// the input key / value pair should be filtered.// - is_changed: if the second return value is true, then it indicates// the existing_value needs to be changed, and the resulting value// is stored in the third return value.// - new_value: if the second return value is true, then this third// return value stores the new value of the input key / value pair.//// -- Examples// -- a filter that keeps all key-value pairs// function Filter(level, key, existing_value)// return false, false, ""// end//// -- a filter that keeps all keys and change their values to "Rocks"// function Filter(level, key, existing_value)// return false, true, "Rocks"// endstd::string lua_script;
一个不使用value(RocksLuaCompactionFilter::ignore_value)的优化,以防您的CompactionFilter从不使用value来决定是保留还是丢弃一个键/值对,然后设置RocksLuaCompactionFilterOptions::ignore_value=true并实现简化的Filter() API。 我们的结果表明,这种优化可以节省高达40%的CPU开销引入LuaCompactionFilter:
// If set to true, then existing_value will not be passed to the Filter// function, and the Filter function only needs to return a single boolean// flag indicating whether to filter out this key or not.//// function Filter(level, key)// ...// return is_filtered// endbool ignore_value = false;
简化的Filter() API只接受两个输入参数,并且只返回一个布尔标志,指示是保留还是放弃输入键。
Error Log 配置 (RocksLuaCompactionFilterOptions::error_log)
* 当RocksLuaCompactionFilter 遇到热河错误时,它将作为no-op执行,并且对于任何导致错误的键/值对总是返回false。开发人员可以配置RocksLuaCompactionFilterOptions::error_log 来记录任何Lua错误:// When specified a non-null pointer, the first "error_limit_per_filter"// errors of each CompactionFilter that is lua related will be included// in this log.std::shared_ptr<Logger> error_log;
注意,对于每个压缩作业,我们只记录error_log中的前几个Lua错误,以避免生成太多的错误消息,并且每个压缩作业报告的错误数量可以通过error_limit_per_filter配置。默认值是1。
// The number of errors per CompactionFilter will be printed// to error_log.int error_limit_per_filter = 1;
动态更新Lua脚本
要在运行RocksDB数据库时更新Lua脚本,只需调用RocksLuaCompactionFilterFactory的SetScript() API:
// Change the Lua script so that the next compaction after this// function call will use the new Lua script.void SetScript(const std::string& new_script);
