Tkrzw
Classes | Public Member Functions | Static Public Member Functions | Static Public Attributes | List of all members
tkrzw::TreeDBM Class Referencefinal

File database manager implementation based on B+ tree. More...

#include <tkrzw_dbm_tree.h>

Classes

class  Iterator
 Iterator for each record. More...
 
struct  TuningParameters
 Tuning parameters for the database. More...
 

Public Member Functions

 TreeDBM ()
 Default constructor. More...
 
 TreeDBM (std::unique_ptr< File > file)
 Constructor with a file object. More...
 
virtual ~TreeDBM ()
 Destructor. More...
 
 TreeDBM (const TreeDBM &rhs)=delete
 Copy and assignment are disabled. More...
 
TreeDBMoperator= (const TreeDBM &rhs)=delete
 
Status Open (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT) override
 Opens a database file. More...
 
Status OpenAdvanced (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT, const TuningParameters &tuning_params=TuningParameters())
 Opens a database file, in an advanced way. More...
 
Status Close () override
 Closes the database file. More...
 
Status Process (std::string_view key, RecordProcessor *proc, bool writable) override
 Processes a record with a processor. More...
 
Status ProcessEach (RecordProcessor *proc, bool writable) override
 Processes each and every record in the database with a processor. More...
 
Status Count (int64_t *count) override
 Gets the number of records. More...
 
Status GetFileSize (int64_t *size) override
 Gets the current file size of the database. More...
 
Status GetFilePath (std::string *path) override
 Gets the path of the database file. More...
 
Status Clear () override
 Removes all records. More...
 
Status Rebuild () override
 Rebuilds the entire database. More...
 
Status RebuildAdvanced (const TuningParameters &tuning_params=TuningParameters())
 Rebuilds the entire database, in an advanced way. More...
 
Status ShouldBeRebuilt (bool *tobe) override
 Checks whether the database should be rebuilt. More...
 
Status Synchronize (bool hard, FileProcessor *proc=nullptr) override
 Synchronizes the content of the database to the file system. More...
 
std::vector< std::pair< std::string, std::string > > Inspect () override
 Inspects the database. More...
 
bool IsOpen () const override
 Checks whether the database is open. More...
 
bool IsWritable () const override
 Checks whether the database is writable. More...
 
bool IsHealthy () const override
 Checks whether the database condition is healthy. More...
 
bool IsOrdered () const override
 Checks whether ordered operations are supported. More...
 
std::unique_ptr< DBM::IteratorMakeIterator () override
 Makes an iterator for each record. More...
 
std::unique_ptr< DBMMakeDBM () const override
 Makes a new DBM object of the same concrete class. More...
 
const FileGetInternalFile () const
 Gets the pointer to the internal file object. More...
 
int64_t GetEffectiveDataSize ()
 Gets the effective data size. More...
 
double GetModificationTime ()
 Gets the last modification time of the database. More...
 
int32_t GetDatabaseType ()
 Gets the database type. More...
 
Status SetDatabaseType (uint32_t db_type)
 Sets the database type. More...
 
std::string GetOpaqueMetadata ()
 Gets the opaque metadata. More...
 
Status SetOpaqueMetadata (const std::string &opaque)
 Sets the opaque metadata. More...
 
KeyComparator GetKeyComparator () const
 Gets the comparator of record keys. More...
 
- Public Member Functions inherited from tkrzw::DBM
virtual ~DBM ()=default
 Destructor. More...
 
virtual Status Process (std::string_view key, RecordLambdaType rec_lambda, bool writable)
 Processes a record with a lambda function. More...
 
virtual Status Get (std::string_view key, std::string *value=nullptr)
 Gets the value of a record of a key. More...
 
virtual std::string GetSimple (std::string_view key, std::string_view default_value="")
 Gets the value of a record of a key, in a simple way. More...
 
virtual std::map< std::string, std::string > GetMulti (const std::initializer_list< std::string > &keys)
 Gets the values of multiple records of keys. More...
 
virtual std::map< std::string, std::string > GetMulti (const std::vector< std::string > &keys)
 Gets the values of multiple records of keys, with a vector. More...
 
virtual Status Set (std::string_view key, std::string_view value, bool overwrite=true, std::string *old_value=nullptr)
 Sets a record of a key and a value. More...
 
virtual Status SetMulti (const std::initializer_list< std::pair< std::string, std::string >> &records, bool overwrite=true)
 Sets multiple records. More...
 
virtual Status SetMulti (const std::map< std::string, std::string > &records, bool overwrite=true)
 Sets multiple records, with a map of strings. More...
 
virtual Status Remove (std::string_view key, std::string *old_value=nullptr)
 Removes a record of a key. More...
 
virtual Status Append (std::string_view key, std::string_view value, std::string_view delim="")
 Appends data at the end of a record of a key. More...
 
virtual Status CompareExchange (std::string_view key, std::string_view expected, std::string_view desired, std::string *actual=nullptr)
 Compares the value of a record and exchanges if the condition meets. More...
 
virtual Status Increment (std::string_view key, int64_t increment=1, int64_t *current=nullptr, int64_t initial=0)
 Increments the numeric value of a record. More...
 
int64_t IncrementSimple (std::string_view key, int64_t increment=1, int64_t initial=0)
 Increments the numeric value of a record, in a simple way. More...
 
virtual Status ProcessEach (RecordLambdaType rec_lambda, bool writable)
 Processes each and every record in the database with a lambda function. More...
 
virtual int64_t CountSimple ()
 Gets the number of records, in a simple way. More...
 
virtual int64_t GetFileSizeSimple ()
 Gets the current file size of the database, in a simple way. More...
 
virtual std::string GetFilePathSimple ()
 Gets the path of the database file, in a simple way. More...
 
virtual bool ShouldBeRebuiltSimple ()
 Checks whether the database should be rebuilt, in a simple way. More...
 
virtual Status CopyFileData (const std::string &dest_path)
 Copies the content of the database file to another file. More...
 
virtual Status Export (DBM *dbm)
 Exports all records to another database. More...
 

Static Public Member Functions

static Status RestoreDatabase (const std::string &old_file_path, const std::string &new_file_path, int64_t end_offset)
 Restores a broken database as a new healthy database. More...
 

Static Public Attributes

static constexpr int32_t DEFAULT_OFFSET_WIDTH = 4
 The default value of the offset width. More...
 
static constexpr int32_t DEFAULT_ALIGN_POW = 10
 The default value of the alignment power. More...
 
static constexpr int64_t DEFAULT_NUM_BUCKETS = 131101
 The default value of the number of buckets. More...
 
static constexpr int32_t DEFAULT_FBP_CAPACITY = 2048
 The default value of the capacity of the free block pool. More...
 
static constexpr int32_t DEFAULT_MAX_PAGE_SIZE = 8130
 The default value of the max page size. More...
 
static constexpr int32_t DEFAULT_MAX_BRANCHES = 256
 The default value of the max branches. More...
 
static constexpr int32_t DEFAULT_MAX_CACHED_PAGES = 10000
 The default value of the maximum number of cached pages. More...
 
static constexpr int32_t OPAQUE_METADATA_SIZE = 10
 The size of the opaque metadata. More...
 

Additional Inherited Members

- Public Types inherited from tkrzw::DBM
typedef std::function< std::string_view(std::string_view, std::string_view)> RecordLambdaType
 Lambda function type to process a record. More...
 

Detailed Description

File database manager implementation based on B+ tree.

All operations are thread-safe; Multiple threads can access the same database concurrently. Every opened database must be closed explicitly to avoid data corruption.

Constructor & Destructor Documentation

◆ TreeDBM() [1/3]

tkrzw::TreeDBM::TreeDBM ( )

Default constructor.

MemoryMapParallelFile is used to handle the data.

◆ TreeDBM() [2/3]

tkrzw::TreeDBM::TreeDBM ( std::unique_ptr< File file)

Constructor with a file object.

Parameters
fileThe file object to handle the data. The ownership is taken.

◆ ~TreeDBM()

virtual tkrzw::TreeDBM::~TreeDBM ( )
virtual

Destructor.

◆ TreeDBM() [3/3]

tkrzw::TreeDBM::TreeDBM ( const TreeDBM rhs)
explicitdelete

Copy and assignment are disabled.

Member Function Documentation

◆ Open()

Status tkrzw::TreeDBM::Open ( const std::string &  path,
bool  writable,
int32_t  options = File::OPEN_DEFAULT 
)
overridevirtual

Opens a database file.

Parameters
pathA path of the file.
writableIf true, the file is writable. If false, it is read-only.
optionsBit-sum options of File::OpenOption enums for opening the file.
Returns
The result status.

Precondition: The database is not opened.

Implements tkrzw::DBM.

◆ OpenAdvanced()

Status tkrzw::TreeDBM::OpenAdvanced ( const std::string &  path,
bool  writable,
int32_t  options = File::OPEN_DEFAULT,
const TuningParameters tuning_params = TuningParameters() 
)

Opens a database file, in an advanced way.

Parameters
pathA path of the file.
writableIf true, the file is writable. If false, it is read-only.
optionsBit-sum options for opening the file.
tuning_paramsA structure for tuning parameters.
Returns
The result status.

Precondition: The database is not opened.

◆ Close()

Status tkrzw::TreeDBM::Close ( )
overridevirtual

Closes the database file.

Returns
The result status.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ Process()

Status tkrzw::TreeDBM::Process ( std::string_view  key,
RecordProcessor proc,
bool  writable 
)
overridevirtual

Processes a record with a processor.

Parameters
keyThe key of the record.
procThe pointer to the processor object.
writableTrue if the processor can edit the record.
Returns
The result status.

Precondition: The database is opened. The writable parameter should be consistent to the open mode.

If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called.

Implements tkrzw::DBM.

◆ ProcessEach()

Status tkrzw::TreeDBM::ProcessEach ( RecordProcessor proc,
bool  writable 
)
overridevirtual

Processes each and every record in the database with a processor.

Parameters
procThe pointer to the processor object.
writableTrue if the processor can edit the record.
Returns
The result status.

Precondition: The database is opened. The writable parameter should be consistent to the open mode.

The ProcessFull of the processor is called repeatedly for each record. The ProcessEmpty of the processor is called once before the iteration and once after the iteration.

Implements tkrzw::DBM.

◆ Count()

Status tkrzw::TreeDBM::Count ( int64_t *  count)
overridevirtual

Gets the number of records.

Parameters
countThe pointer to an integer object to contain the result count.
Returns
The result status.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ GetFileSize()

Status tkrzw::TreeDBM::GetFileSize ( int64_t *  size)
overridevirtual

Gets the current file size of the database.

Parameters
sizeThe pointer to an integer object to contain the result size.
Returns
The result status.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ GetFilePath()

Status tkrzw::TreeDBM::GetFilePath ( std::string *  path)
overridevirtual

Gets the path of the database file.

Parameters
pathThe pointer to a string object to contain the result path.
Returns
The result status.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ Clear()

Status tkrzw::TreeDBM::Clear ( )
overridevirtual

Removes all records.

Returns
The result status.

Precondition: The database is opened as writable.

Implements tkrzw::DBM.

◆ Rebuild()

Status tkrzw::TreeDBM::Rebuild ( )
overridevirtual

Rebuilds the entire database.

Returns
The result status.

Precondition: The database is opened as writable.

Rebuilding a database is useful to reduce the size of the file by solving fragmentation. All tuning parameters are succeeded or calculated implicitly.

Implements tkrzw::DBM.

◆ RebuildAdvanced()

Status tkrzw::TreeDBM::RebuildAdvanced ( const TuningParameters tuning_params = TuningParameters())

Rebuilds the entire database, in an advanced way.

Parameters
tuning_paramsA structure for tuning parameters. The default value of each parameter means that the current setting is succeeded or calculated implicitly.
Returns
The result status.

Precondition: The database is opened as writable.

Rebuilding a database is useful to reduce the size of the file by solving fragmentation. Tuning parameters for the underlying hash database are reflected on the rebuilt file on the spot. Tuning parameters for B+ tree are reflected gradually while updating the database later. The comparator of record keys cannot be changed.

◆ ShouldBeRebuilt()

Status tkrzw::TreeDBM::ShouldBeRebuilt ( bool *  tobe)
overridevirtual

Checks whether the database should be rebuilt.

Parameters
tobeThe pointer to a boolean object to contain the result decision.
Returns
The result status.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ Synchronize()

Status tkrzw::TreeDBM::Synchronize ( bool  hard,
FileProcessor proc = nullptr 
)
overridevirtual

Synchronizes the content of the database to the file system.

Parameters
hardTrue to do physical synchronization with the hardware or false to do only logical synchronization with the file system.
procThe pointer to the file processor object, whose Process method is called while the content of the file is synchronized. If it is nullptr, it is ignored.
Returns
The result status.

Precondition: The database is opened as writable.

Implements tkrzw::DBM.

◆ Inspect()

std::vector<std::pair<std::string, std::string> > tkrzw::TreeDBM::Inspect ( )
overridevirtual

Inspects the database.

Returns
A vector of pairs of a property name and its value.

Implements tkrzw::DBM.

◆ IsOpen()

bool tkrzw::TreeDBM::IsOpen ( ) const
overridevirtual

Checks whether the database is open.

Returns
True if the database is open, or false if not.

Implements tkrzw::DBM.

◆ IsWritable()

bool tkrzw::TreeDBM::IsWritable ( ) const
overridevirtual

Checks whether the database is writable.

Returns
True if the database is writable, or false if not.

Implements tkrzw::DBM.

◆ IsHealthy()

bool tkrzw::TreeDBM::IsHealthy ( ) const
overridevirtual

Checks whether the database condition is healthy.

Returns
True if the database condition is healthy, or false if not.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ IsOrdered()

bool tkrzw::TreeDBM::IsOrdered ( ) const
overridevirtual

Checks whether ordered operations are supported.

Returns
Always true. Ordered operations are supported.

Implements tkrzw::DBM.

◆ MakeIterator()

std::unique_ptr<DBM::Iterator> tkrzw::TreeDBM::MakeIterator ( )
overridevirtual

Makes an iterator for each record.

Returns
The iterator for each record.

Precondition: The database is opened.

Implements tkrzw::DBM.

◆ MakeDBM()

std::unique_ptr<DBM> tkrzw::TreeDBM::MakeDBM ( ) const
overridevirtual

Makes a new DBM object of the same concrete class.

Returns
The new file object.

Implements tkrzw::DBM.

◆ GetInternalFile()

const File* tkrzw::TreeDBM::GetInternalFile ( ) const

Gets the pointer to the internal file object.

Returns
The pointer to the internal file object.

Accessing the internal file viorates encapsulation policy. This should be used only for testing and debugging.

◆ GetEffectiveDataSize()

int64_t tkrzw::TreeDBM::GetEffectiveDataSize ( )

Gets the effective data size.

Returns
The effective data size, or -1 on failure.

Precondition: The database is opened.

The effective data size means the total size of the keys and the values.

◆ GetModificationTime()

double tkrzw::TreeDBM::GetModificationTime ( )

Gets the last modification time of the database.

Returns
The last modification time of the UNIX epoch, or -1 on failure.

Precondition: The database is opened.

◆ GetDatabaseType()

int32_t tkrzw::TreeDBM::GetDatabaseType ( )

Gets the database type.

Returns
The database type, or -1 on failure.

Precondition: The database is opened.

◆ SetDatabaseType()

Status tkrzw::TreeDBM::SetDatabaseType ( uint32_t  db_type)

Sets the database type.

Parameters
db_typeThe database type.
Returns
The result status.

Precondition: The database is opened as writable.

This data is just for applications and not used by the database implementation.

◆ GetOpaqueMetadata()

std::string tkrzw::TreeDBM::GetOpaqueMetadata ( )

Gets the opaque metadata.

Returns
The opaque metadata, or an empty string on failure.

Precondition: The database is opened.

◆ SetOpaqueMetadata()

Status tkrzw::TreeDBM::SetOpaqueMetadata ( const std::string &  opaque)

Sets the opaque metadata.

Parameters
opaqueThe opaque metadata, of which leading 16 bytes are stored in the file.
Returns
The result status.

Precondition: The database is opened as writable.

This data is just for applications and not used by the database implementation.

◆ GetKeyComparator()

KeyComparator tkrzw::TreeDBM::GetKeyComparator ( ) const

Gets the comparator of record keys.

Returns
the key comparator function, or nullptr on failure.

Precondition: The database is opened.

◆ RestoreDatabase()

static Status tkrzw::TreeDBM::RestoreDatabase ( const std::string &  old_file_path,
const std::string &  new_file_path,
int64_t  end_offset 
)
static

Restores a broken database as a new healthy database.

Parameters
old_file_pathThe path of the broken database.
new_file_pathThe path of the new database to be created.
end_offsetThe exclusive end offset of records to read. Negative means unlimited. 0 means the size when the database is synched or closed properly.
Returns
The result status.

Member Data Documentation

◆ DEFAULT_OFFSET_WIDTH

constexpr int32_t tkrzw::TreeDBM::DEFAULT_OFFSET_WIDTH = 4
static

The default value of the offset width.

◆ DEFAULT_ALIGN_POW

constexpr int32_t tkrzw::TreeDBM::DEFAULT_ALIGN_POW = 10
static

The default value of the alignment power.

◆ DEFAULT_NUM_BUCKETS

constexpr int64_t tkrzw::TreeDBM::DEFAULT_NUM_BUCKETS = 131101
static

The default value of the number of buckets.

◆ DEFAULT_FBP_CAPACITY

constexpr int32_t tkrzw::TreeDBM::DEFAULT_FBP_CAPACITY = 2048
static

The default value of the capacity of the free block pool.

◆ DEFAULT_MAX_PAGE_SIZE

constexpr int32_t tkrzw::TreeDBM::DEFAULT_MAX_PAGE_SIZE = 8130
static

The default value of the max page size.

◆ DEFAULT_MAX_BRANCHES

constexpr int32_t tkrzw::TreeDBM::DEFAULT_MAX_BRANCHES = 256
static

The default value of the max branches.

◆ DEFAULT_MAX_CACHED_PAGES

constexpr int32_t tkrzw::TreeDBM::DEFAULT_MAX_CACHED_PAGES = 10000
static

The default value of the maximum number of cached pages.

◆ OPAQUE_METADATA_SIZE

constexpr int32_t tkrzw::TreeDBM::OPAQUE_METADATA_SIZE = 10
static

The size of the opaque metadata.