DMLite: How to use it as a backend in C LOGO

Introduction

Although DMLite is written in C++, it has a built-in wrapper in C, which allows to use the library on C applications.

Initialize the library

We will start with a simple appplication skeleton which does nothing.

#include <stdio.h>

int main(int argc, char *argv[])
{
  return 0;
}

Now, we want to start using DMLite, so first of all we need to include the headers and instantiate a dmlite_manager, which controls the loading and instantiation process of a plugin stack. Then, we will request the manager to use the configuration file specified as the first parameter of the application.

#include <stdio.h>
#include <dmlite/c/dmlite.h>

int main(int argc, char *argv[])
{
  dmlite_manager *manager;
  
  if (argc < 2) {
    printf("Need at least one parameter.\n");
    return 1;
  }
  
  manager = dmlite_manager_new();
  if (dmlite_manager_load_configuration(manager, argv[1]) != 0) {
    printf("Could not load the configuration file\n");
    printf("Reason: %s\n", dmlite_manager_error(manager));
    return 2;
  }
  
  dmlite_manager_free(manager);
  return 0;
}

Now you can compile and try it.

# gcc main.c -ldmlite -o dmlite-init
# ./dmlite-init /etc/init.conf

Stat of a file

Initialize a context

To actually start to perform actions on the namespace, we need to instantiate a dmlite_context before, and to set a security context, either directly, or passing some credentials first.

#include <stdio.h>
#include <dmlite/c/dmlite.h>
#include <dmlite/c/catalog.h>
#include <string.h>

int main(int argc, char *argv[])
{
  dmlite_manager *manager;
  dmlite_context *context;
  dmlite_credentials creds;
  int retCode = 0;

  if (argc < 3) {
    printf("Usage: %s <configuration file> <path>\n", argv[0]);
    return 1;
  }

  /* Create manager */
  manager = dmlite_manager_new();
  if (dmlite_manager_load_configuration(manager, argv[1]) != 0) {
    printf("Could not load the configuration file\n");
    printf("Reason: %s\n", dmlite_manager_error(manager));
    return 2;
  }

  /* Create context */
  context = dmlite_context_new(manager);
  if (context == NULL) {
    printf("Could not instantiate a context.\n");
    printf("Reason: %s\n", dmlite_manager_error(manager));
    return 3;
  }

  /* Set security credentials */
  memset(&creds, 0, sizeof(dmlite_credentials));
  creds.client_name    = "/C=CH/O=CERN/OU=GD/CN=Test user 1";
  creds.remote_address = "127.0.0.1";

  if (dmlite_setcredentials(context, &creds) != 0) {
    printf("Could not set the credentials.\n");
    printf("Reason: %s\n", dmlite_error(context));
    return 4;
  }
  
  /* Action */

  /* Clean */
  dmlite_context_free(context);
  dmlite_manager_free(manager);
  return retCode;
}

Please, note that the client name you are setting must be defined in the file /etc/lcgdm-mapfile, so a default group can be picked. You can always initialize the field fqans and nfqans to contain at least one group (i.e. dteam)

Perform the action

Let's add the code that is going to actually do something.

  /* Action */
  struct stat fstat;

  if (dmlite_stat(context, argv[2], &fstat) == 0) {
    printf("File owner: %u\n", fstat.st_uid);
    printf("File group: %u\n", fstat.st_gid);
    printf("File mode:  %4o\n", fstat.st_mode);
  }
  else {
    printf("Could not stat the file.\n");
    printf("Reason: %s\n", dmlite_error(context));
    retCode = 4;
  }

You should get this output now:

# ./dmlite-init /etc/dmlite.conf /
File owner: 0
File group: 0
File mode:  40755
# ./dmlite-init /etc/dmlite.conf /xx
Could not stat the file.
Reason: [#00.000002] Entry 'xx' not found under ''

Note: Check the function dmlite_statx to see how to retrieve extended attributes (as ACL, GUID, etc...)

List a directory

For this example, we will reuse the base code as shown just before. Again, it should be quite easy to follow if you are familiar with the POSIX calls opendir/readdir/closedir. Remember to add the header dirent.h, so you can derreference the struct dirent* pointer.

/* Action */
dmlite_dir *dir;
dir = dmlite_opendir(context, argv[2]);
if (dir == NULL) {
  printf("Could not open the directory.\n");
  printf("Reason: %s\n", dmlite_error(context);
  retCode = 4;
}
else {
  struct dirent *e;
  printf("Content of the directory %s:\n", argv[2]);
  while ((e = dmlite_readdir(context, dir)) != NULL) {
    printf("\t%s\n", e->d_name);
  }
  dmlite_closedir(context, dir);
}

The output this time should be:

# ./dmlite-init /etc/dmlite.conf /xx
Could not open the directory.
Reason: [#00.000002] Entry 'xx' not found under ''
# ./dmlite-init /etc/dmlite.conf /
Content of the directory /:
	dpm

There is another method called dmlite_readdirx, which allows to access directly to all the metadadata, as it would return dmlite_statx.

Get a physical location where to read

This use case is a little bit trickier, since replicas do not necessarily point to a physical file somewhere. It may be the case, but you should not rely on it, since some translation may be needed (i.e. for Hadoop or S3 pools).
However, for convenience there is one method called dmlite_get, which will use the C++ API to get one single replica and translate it. Keep in mind that it actually returns an array of physical location, since the API has support for pool types that may split the file between multiple nodes (as Hadoop does).
Remember to add the header file dmlite/c/pool.h

/* Action */
dmlite_location *location;

location = dmlite_get(context, argv[2]);
if (location == NULL) {
  printf("Could not get the final location for %s\n", argv[2]);
  printf("Reason: %s\n", dmlite_error(context));
  retCode = 4;
}
else {
  int i;
  
  for (i = 0; i < location->nchunks; ++i) {
    printf("Chunk %d: %s:%s (%lu-%lu)\n", i,
           location->chunks[i].host, location->chunks[i].path,
           location->chunks[i].offset,
           location->chunks[i].offset + location->chunks[i].size);
    
    if (location->chunks[i].extra != NULL) {
      int j, nKeys;
      char **keys;
      char buffer[128];
      
      dmlite_any_dict_keys(location->chunks[i].extra, &nKeys, &keys);
      for (j = 0; j < nKeys; ++j) {
        dmlite_any *any = dmlite_dict_get(location->chunks[i].extra, keys[j]);
        dmlite_any_to_string(any, buffer, sizeof(buffer));
        printf("\t%s: %s\n", keys[j], buffer);
        dmlite_any_free(any);
      }
      dmlite_any_dict_keys_free(nKeys, keys);
    }
  }
  
  dmlite_location_free(context, location);
}

The output will depend on what you have on your DPM server, but the output should look something like this:

# ./dmlite-init /etc/dmlite.conf /dpm/cern.ch/home/dteam/file.html
Chunk 0: sl6vm.cern.ch:/dpm/cern.ch/home/dteam/dmlite03_test/test529.html (0-63214)
	token: NwwcFOCZDUjPCccmrNulbyckWTQ=@1348067073@0

Now, this gives the physical location of the replica, but it is not tied to any protocol in particular. The front-end is the responsible of determining which protocol should be used to access the actual data.

For writing, dmlite_put works in the same manner, but a dmlite_donewriting must be done when the writing is finished.

Read the file

There is one limitation of DMLite design, which is that only one I/O plugin can be used at a time. This is because in principle each node belongs to one single pool. This is, one node can not belong to a Hadoop pool and to a Filesytem pool at the same time.

To open a file four things are needed: a context, a physical path, an open mode (read, write, ...) and whatever extra information the corresponding pool plugin gave us. For instance, in the previous example we show that extra information, stored as key/value pairs. Usually this data includes the necessary information to validate the access (i.e. a token).
Once the file is opened, usual read and/or write operations (and also seek/tell/eof/...) can be performed.

The following example assumes you have already deserialized the additional information coming from the head node, and put it into a dmlite_any_dict.

#include <dmlite/c/io.h>
#include <stdio.h>

int dump_data(dmlite_context *context, const char *path, dmlite_any_dict *args, FILE *out)
{
  dmlite_fd *in;
  char buffer[1024];
  size_t size;
  
  in = dmlite_fopen(context, path, O_RDONLY, args);
  if (in == NULL) {
    fprintf(stderr, "Could not open the input file: %s\n", dmlite_error(context));
    return dmlite_errno(context);
  }
  
  while (!dmlite_feof(in)) {
    size = dmlite_fread(in, buffer, sizeof(buffer));
    fwrite(buffer, sizeof(char), size, out);
  }
  
  dmlite_fclose(in);
}

As you can see, except dmlite_fopen, the rest of the functions have a signature similar to the standard ones, making the adaptation of existing code easier.