从RocksDB 6.0开始弃用和删除
RocksDB CompactionFilter提供了一种基于后台自定义逻辑删除/过滤过期键/值对的方法。 现在我们在它上面实现了一个扩展,允许用户在Lua中实现自定义的CompactionFilter。这个特性在RocksDB 5.0中可用。
好处
在Lua中开发CompactionFilter有以下好处:
动态更改: 可以在不关闭RocksDB实例的情况下动态更新Lua压缩过滤器。
单个二进制单元: 由于可以动态地更新Lua压缩过滤器,因此不再需要按照C++压缩过滤器的要求重新构建二进制。
这对于构建在RocksDB之上的服务来说是一个巨大的好处,RocksDB为客户维护定制的CompactionFilters。
更安全: CompactionFilter中的错误将不再导致核心转储。Lua 引擎捕获所有异常。
如何使用?
使用RocksLuaCompactionFilter很简单。你需要做的就是以下步骤:
将LUA_PATH设置为Lua的根目录,构建RocksDB。
用lua脚本配置RocksLuaCompactionFilterOptions。(更多细节将在下一节描述)
使用步骤1中的RocksLuaCompactionFilterOptions构建了一个 RocksLuaCompactionFilterFactory。
将步骤2中的RocksLuaCompactionFilterFactory传递到ColumnFamilyOptions::compaction_filter_factory。
例子
这里面有一个简单的RocksLuaCompactionFilter,过滤掉任何键的初始值小于r:
lua::RocksLuaCompactionFilterOptions lua_opt; // removes all keys whose initial is less than ‘r’ lua_opt.lua_script =
"function Filter(level, key, existing_value)\n"
" if key:sub(1,1) < 'r' then\n"
" return true, false, \"\"\n"
" end\n"
" return false, false, \"\"\n"
"end\n"
"\n"
"function FilterMergeOperand(level, key, operand)\n"
" return false\n"
"end\n"
"function Name()\n"
" return \"KeepsAll\"\n"
"end\n";
// specify error log. auto* rocks_logger = new facebook::rocks::RocksLogger(
"RocksLuaTest",
true, // print error message in GLOG
true, // print error message to scribe, available in LogView `RocksDB ERROR`
nullptr);
// Create RocksLuaCompactionFilter with the above lua_opt and pass it to Options rocksdb::Options options; options.compaction_filter_factory =
std::make_shared<rocksdb::lua::RocksLuaCompactionFilterFactory>(lua_opt);
… // open FbRocksDB with the above options rocksdb::DB* db; auto status = openRocksDB(options, “RocksDBWithLua”, &db);
配置 RocksLuaCompactionFilterOptions
在这里,我们将更详细地介绍如何配置RocksLuaCompactionFilterOptions。
RocksLuaCompactionFilterOptions的定义可以在include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h中找到。
配置Lua脚本(RocksLuaCompactionFilter::Script)
第一个也是最重要的参数是RocksLuaCompactionFilterOptions::script,您的Lua压缩过滤器将在这里实现。Lua脚本必须实现所需的函数,即Name()和Filter()。
// The lua script in string that implements all necessary CompactionFilter
// virtual functions. The specified lua_script must implement the following
// functions, which are Name and Filter, as described below.
//
// 0. The Name function simply returns a string representing the name of
// the lua script. If there's any erorr in the Name function, an
// empty string will be used.
// --- Example
// function Name()
// return "DefaultLuaCompactionFilter"
// end
//
//
// 1. The script must contains a function called Filter, which implements
// CompactionFilter::Filter() , takes three input arguments, and returns
// three values as the following API:
//
// function Filter(level, key, existing_value)
// ...
// return is_filtered, is_changed, new_value
// end
//
// Note that if ignore_value is set to true, then Filter should implement
// the following API:
//
// function Filter(level, key)
// ...
// return is_filtered
// end
//
// If there're any error in the Filter() function, then it will keep
// the input key / value pair.
//
// -- Input
// The function must take three arguments (integer, string, string),
// which map to "level", "key", and "existing_value" passed from
// RocksDB.
//
// -- Output
// The function must return three values (boolean, boolean, string).
// - is_filtered: if the first return value is true, then it indicates
// the input key / value pair should be filtered.
// - is_changed: if the second return value is true, then it indicates
// the existing_value needs to be changed, and the resulting value
// is stored in the third return value.
// - new_value: if the second return value is true, then this third
// return value stores the new value of the input key / value pair.
//
// -- Examples
// -- a filter that keeps all key-value pairs
// function Filter(level, key, existing_value)
// return false, false, ""
// end
//
// -- a filter that keeps all keys and change their values to "Rocks"
// function Filter(level, key, existing_value)
// return false, true, "Rocks"
// end
std::string lua_script;
一个不使用value(RocksLuaCompactionFilter::ignore_value)的优化,以防您的CompactionFilter从不使用value来决定是保留还是丢弃一个键/值对,然后设置RocksLuaCompactionFilterOptions::ignore_value=true并实现简化的Filter() API。 我们的结果表明,这种优化可以节省高达40%的CPU开销引入LuaCompactionFilter:
// If set to true, then existing_value will not be passed to the Filter
// function, and the Filter function only needs to return a single boolean
// flag indicating whether to filter out this key or not.
//
// function Filter(level, key)
// ...
// return is_filtered
// end
bool ignore_value = false;
简化的Filter() API只接受两个输入参数,并且只返回一个布尔标志,指示是保留还是放弃输入键。
Error Log 配置 (RocksLuaCompactionFilterOptions::error_log)
* 当RocksLuaCompactionFilter 遇到热河错误时,它将作为no-op执行,并且对于任何导致错误的键/值对总是返回false。开发人员可以配置RocksLuaCompactionFilterOptions::error_log 来记录任何Lua错误:
// When specified a non-null pointer, the first "error_limit_per_filter"
// errors of each CompactionFilter that is lua related will be included
// in this log.
std::shared_ptr<Logger> error_log;
注意,对于每个压缩作业,我们只记录error_log中的前几个Lua错误,以避免生成太多的错误消息,并且每个压缩作业报告的错误数量可以通过error_limit_per_filter配置。默认值是1。
// The number of errors per CompactionFilter will be printed
// to error_log.
int error_limit_per_filter = 1;
动态更新Lua脚本
要在运行RocksDB数据库时更新Lua脚本,只需调用RocksLuaCompactionFilterFactory的SetScript() API:
// Change the Lua script so that the next compaction after this
// function call will use the new Lua script.
void SetScript(const std::string& new_script);