DMLite: How to use it as a backend in C++ LOGO

Introduction

These examples are the same as those explained using the C wrapper, but in this case, the C++ API is directly used.

Initialize the library

We will start with a simple appplication skeleton which does nothing.

#include <iostream>

int main(int argc, char *argv[])
{
  return 0;
}

Now, we want to start using DMLite, so first of all we need to include the headers and instantiate a dmlite::PluginManager, which controls the loading and instantiation process of a plugin stack. Then, we will request the manager to use the configuration file specified as the first parameter of the application.

#include <iostream>
#include <dmlite/cpp/dmlite.h>

int main(int argc, char *argv[])
{
  dmlite::PluginManager manager;
  
  if (argc < 2) {
    std::cout << "Need at least one parameter." std::endl;
    return 1;
  }
  
  try {
    manager.loadConfiguration(argv[1]);
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not load the configuration file." << std::endl
              << "Reason: " << e.what() << std::endl;
    return 1;
  }

  return 0;
}

Now you can compile and try it.

# gcc main.cpp -ldmlite -o dmlite-init
# ./dmlite-init /etc/init.conf

Stat of a file

Initialize a context

To actually start to perform actions on the namespace, we need to instantiate a dmlite::StackInstance before, and to set a security context, either directly, or passing some credentials first.

#include <iostream>
#include <dmlite/cpp/authn.h>
#include <dmlite/cpp/dmlite.h>
#include <dmlite/cpp/catalog.h>

int main(int argc, char *argv[])
{
  dmlite::PluginManager manager;
  
  if (argc < 3) {
    std::cout << "Usage: " << argv[0]
              << " <configuration file> <path>" << std::endl;
    return 1;
  }
  
  // Create manager
  try {
    manager.loadConfiguration(argv[1]);
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not load the configuration file." << std::endl
              << "Reason: " << e.what() << std::endl;
    return 2;
  }

  // Create StackInstance
  dmlite::StackInstance stack(&manager);

  // Set security credentials
  dmlite::SecurityCredentials creds;
  creds.clientName = "/C=CH/O=CERN/OU=GD/CN=Test user 1";
  creds.remoteAddress = "127.0.0.1";
  try {
    stack.setSecurityCredentials(creds);
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not set the credentials." << std::endl
              << "Reason: " << e.what() << std::endl;
    return 4;
  }

  // Action

  // Done
  return 0;
}

Please, note that the client name you are setting must be defined in the file /etc/lcgdm-mapfile, so a default group can be picked. You can always initialize the field fqans to contain at least one group (i.e. dteam)

One advantage of the C++ API is that you don't need to take care of freeing resources everytime an error occurs. Since PluginManager and StackInstance can be used as objects, and not as pointers, they will be automatically destroyed as soon as they fall out of scope.

Perform the action

Let's add the code that is going to actually do something.

  // Action
  dmlite::Catalog* catalog = stack.getCatalog();

  try {
    dmlite::ExtendedStat xstat = catalog->extendedStat(argv[2]);
    std::cout << "File name:  " << xstat.name << std::endl
              << "File owner: " << xstat.stat.st_uid << std::endl
              << "File group: " << xstat.stat.st_gid << std::endl
              << "File mode:  " << std::oct << xstat.stat.st_mode << std::endl;
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not stat the file." << std::endl
              << "Reason: " << e.what() << std::endl;
    return e.code();
  }

You should get this output now:

# ./dmlite-init /etc/dmlite.conf /
File name:  /
File owner: 0
File group: 0
File mode:  40755
# ./dmlite-init /etc/dmlite.conf /xx
Could not stat the file.
Reason: [#00.000002] Entry 'xx' not found under ''

List a directory

For this example, we will reuse the base code as shown just before.

  // Action
  dmlite::Catalog* catalog = stack.getCatalog();

  try {
    dmlite::Directory* dir = catalog->openDir(argv[2]);
    dmlite::ExtendedStat* xstat;

    std::cout << "Content of the directory " << argv[2] << std::endl;
    while ((xstat = catalog->readDirx(dir)) != NULL) {
      std::cout << '\t' << xstat->name << std::endl;
    }

    catalog->closeDir(dir);
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not open the directory." << std::endl
              << "Reason: " << e.what() << std::endl;
    return e.code();
  }

The output this time should be:

# ./dmlite-init /etc/dmlite.conf /xx
Could not open the directory.
Reason: [#00.000002] Entry 'xx' not found under ''
# ./dmlite-init /etc/dmlite.conf /
Content of the directory /
	dpm

Get a physical location where to read

This use case is a little bit trickier, since replicas do not necessarily point to a physical file somewhere. It may be the case, but you should not rely on it, since some translation may be needed (i.e. for Hadoop or S3 pools).

For this example, you will need to include the header dmlite/cpp/poolmanager.h

  dmlite::PoolManager* poolManager = stack.getPoolManager();

  try {
    dmlite::Location location = poolManager->whereToRead(argv[2]);
    dmlite::Location::const_iterator i;
    int n;
    
    for (i = location.begin(), n = 0; i != location.end(); ++i, ++n) {
      std::cout << "Chunk " << n << ": "
                << i->host << ':' << i->path << " ("
                << i->offset << '-' << i->offset + i->size << ")"
                << std::endl;

      std::vector<std::string> keys = i->getKeys();
      std::vector<std::string>::const_iterator j;

      for (j = keys.begin(); j != keys.end(); ++j) {
        std::cout << '\t' << *j << ": " << i->getString(*j) << std::endl;
      }
    }
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not get the final location for "
              << argv[2] << std::endl
              << "Reason: " << e.what() << std::endl;
    return e.code();
  }

The output will depend on what you have on your DPM server, but the output should look something like this:

# ./dmlite-init /etc/dmlite.conf /dpm/cern.ch/home/dteam/file.html
Chunk 0: sl6vm.cern.ch:/dpm/cern.ch/home/dteam/dmlite03_test/test529.html (0-63214)
	token: NwwcFOCZDUjPCccmrNulbyckWTQ=@1348067073@0

Now, this gives the physical location of the replica, but it is not tied to any protocol in particular. The front-end is the responsible of determining which protocol should be used to access the actual data.

For writing, whereToWrite works in a similar manner, but a call to doneWriting must be done when the writing finishes.

Read the file

There is one limitation of DMLite design, which is that only one I/O plugin can be used at a time. This is because in principle each node belongs to one single pool. This is, one node can not belong to a Hadoop pool and to a Filesytem pool at the same time.

To open a file four things are needed: a stack instance, a physical path, an open mode (read, write, ...) and whatever extra information the corresponding pool plugin gave us. For instance, in the previous example we show that extra information, stored as key/value pairs. Usually this data includes the necessary information to validate the access (i.e. a token).
Once the file is opened, usual read and/or write operations (and also seek/tell/eof/...) can be performed.

The following example assumes you have already deserialized the additional information coming from the head node, and put it into an dmlite::Extensible object.

#include <dmlite/cpp/io.h>
#include <iostream>

int dump_data(dmlite::StackInstance& stack, const char *path,
              const dmlite::Extensible& args, std::ostream& out)
{
  char buffer[1024];
  size_t size;
  dmlite::IODriver* iodriver;

  iodriver = stack.getIODriver();
  
  try {
    dmlite::IOHandler* in = iodriver->createIOHandler(path, O_RDONLY, args)
    
    while (!in->eof()) {
      size = in->read(buffer, sizeof(buffer));
      out.write(buffer, size);
    }
    
    delete in;
  }
  catch (dmlite::DmException& e) {
    std::cout << "Could not open the input file: " << path << std::endl;
    std::cout << "Reason: " << e.what() << std::endl;
    return e.code();
  }
  
  return 0;
}