pdbrep.nw

Representation of the generic PDB information

Representation of the generic PDB information

Interface

Database header
Record header
Commonly used category information

Implementation
Time conversions

The unpacking step

Interface

We begin with the I/O functions. read_db reads in an entire database from a file. Some pieces are uninterpreted Text and must be dealt with later, by specialized converter modules. write_dbinfo writes out only the informational header, and write_record_header writes out a record header.

<header>= (U->) [D->]
#include "pdbio.h"
#include <time.h>
struct DBInfo; /* defined below */
struct record; /* defined below */
extern struct pdb read_db (FILE *fp); /* reads file, fills in data structures */
extern int write_dbinfo (FILE *fp, struct DBInfo *info,
                         unsigned long appoff, unsigned long sortoff);
         /* writes file, returns # of bytes written */
extern void write_record_header(FILE *fp, struct record *rec, unsigned offset);

Database header

Here is the standard database information. The appInfo and sortInfo fields, as well as the records, are application-specific, so they are represented here as arbitrary strings of characters (see pdbio.nw for the definition of Text).

<header>+= (U->) [<-D->]
struct DBInfo {
  char name[33];                /* name of database, plus null */
  unsigned resource:1;          /* resource DB, not record db */
  unsigned readonly:1;
  unsigned dirtyAppInfo:1;
  unsigned backup:1;    /* no conduit exists; use standard backup */
  unsigned installOver:1;       /* ok to install over open DB */
  unsigned reset:1;             /* force pilot to reset after installing */
  unsigned version;             /* application-dependent */
  time_t createDate,modifyDate,backupDate;
  unsigned long modnum;         /* modification number */
  Text appInfo;
  Text sortInfo;
  char type[5];                 /* 4-byte type, plus trailing null */
  char creator[5];              /* 4-byte creator ID, plus trailing null */
  unsigned long uidSeed;        /* seed for creating unique record IDs */
  unsigned long nextRecordListID; /* set to zero? */
  unsigned short numrecs;       /* number of records */
};

The entire database is laid out in the file like this. The role of the filler is a mystery, but we include it here so we can read a binary PDB file, convert it, and write back the original unchanged---it's a good test.

<header>+= (U->) [<-D->]
typedef struct pdb {
  struct DBInfo info;
  unsigned short filler;
  struct record *records;       /* list of all the records */
  Text image;                   /* the whole file as it appeared on disk */
} DB;

Defines DB (links are to index).

Record header

Each record has a category and some flag bits in addition to its application-dependent data.

<header>+= (U->) [<-D->]
typedef struct record {
  Text data;
  unsigned secret:1;
  unsigned busy:1;
  unsigned dirty:1;
  unsigned delete:1;
  unsigned category:4;
  unsigned uid:24;              /* 24-bit unique id -- zero for new recs */
} Record;

Defines Record (links are to index).

Commonly used category information

Finally, many applications store the same kind of category information, so we provide hooks for it here:

<header>+= (U->) [<-D->]
struct PDBCategoryAppInfo {
  unsigned int renamed[16]; /* Boolean array of categories with changed names */
  char name[16][16]; /* 16 categories of 15 characters+nul each */
  unsigned char ID[16]; 
  unsigned char lastUniqueID; /* Each category gets a unique ID, for sync tracking
                                 purposes. Those from the Pilot are between 0 & 127.
                                 Those from the PC are between 128 & 255. I'm not
                                 sure what role lastUniqueID plays. */
};

We provide read, write, pack, and unpack functions for the category stuff.

<header>+= (U->) [<-D]
extern struct PDBCategoryAppInfo catread(Text *);
extern void unpack_categories(FILE *out, struct PDBCategoryAppInfo *cat, int cols);
struct PDBCategoryAppInfo pack_categories(void);
int catwrite(FILE *fp, struct PDBCategoryAppInfo *ai);

Implementation

A lot of this code was written based on close reading of the most excellent ``Pilot-Link'' package written by Kenneth Albanowski. The time-conversion functions were taken directly.

<*>=
#include <stdlib.h>
#include <assert.h>
#include "pushchar.h"
#include "pdblua.h"
<header>
<functions>

Time conversions

These conversion functions apply no timezone correction. UNIX uses UTC for time_t's, while the Pilot uses local time for database backup time and appointments, etc. It is not particularly simple to convert between these in UNIX, especially since the Pilot's local time is unknown, and if syncing over political boundries, could easily be different then the local time on the UNIX box. Since the Pilot does not know what timezone it is in, there is no unambiguous way to correct for this.

Worse, the creation date for a program is stored in the local time of the computer which did the final linking of that program. Again, the Pilot does not store the timezone information needed to reconstruct where/when this was.

A better immediate tack would be to dissect these into struct tm's, and return those.

<functions>= (<-U) [D->]
/* Exact value of "Jan 1, 1970 0:00:00 GMT" - "Jan 1, 1904 0:00:00 GMT" */
#define PILOT_TIME_DELTA (unsigned)(2082844800)
   
static time_t pilot_time_to_unix_time (unsigned long raw_time) {
  return (time_t)(raw_time - PILOT_TIME_DELTA);
}

static unsigned long unix_time_to_pilot_time (time_t t) {
  return (unsigned long)((unsigned long)t + PILOT_TIME_DELTA);
}

Defines PILOT_TIME_DELTA, time_t, unix_time_to_pilot_time (links are to index).

The unpacking step

These are the standard flags.

<functions>+= (<-U) [<-D->]
enum dbflags { 
  Fresource=0x0001,
  Freadonly=0x0002, FdirtyAppInfo=0x0004, Fbackup=0x0008,
  FinstallOver=0x0010, Freset=0x0020,
  Fopen=0x8000
};
enum recordflags {
  Rsecret = 0x10, Rbusy = 0x20, Rdirty = 0x40, Rdelete = 0x80
};

While reading, we do some sanity checks on the offsets.

<functions>+= (<-U) [<-D->]
static void checkoffset(struct pdb *db, unsigned long offset, char *file, int line) {
  if ((int)offset < 0 || offset > db->image.len) {
    fprintf(stderr, "%s:%d: Bad offset %lu (swapped %lu, hex %lx); limit is %d\n",
            file, line,
            offset, 
            ((offset & 0xff) << 24 | (offset & 0xff00 << 8) | 
             (offset & 0xff0000) >> 8 | (offset & 0xff000000 >> 24)),
            offset, db->image.len);
    abort();
  }
}
#define checkoffset(x, y) checkoffset(x, y, __FILE__, __LINE__)

Defines checkoffset (links are to index).

This code is brainless. Just picky decoding.

<functions>+= (<-U) [<-D->]
struct pdb read_db (FILE *fp) {
  struct pdb db;
  struct DBInfo *ip = &db.info;
  Text *t;
  unsigned long app_info_offset, sort_info_offset;
  unsigned flags;
  int i;

  db.image = readfile(fp);
  t = &db.image;

  get_chars(ip->name, t, 32);
  ip->name[32] = 0;
  flags = get_short(t);
  ip->resource = flags & Fresource ? 1 : 0;
  ip->readonly = flags & Freadonly ? 1 : 0;
  ip->dirtyAppInfo = flags & FdirtyAppInfo ? 1 : 0;
  ip->backup = flags & Fbackup ? 1 : 0;
  ip->installOver = flags & FinstallOver ? 1 : 0;
  ip->reset = flags & Freset ? 1 : 0;
  ip->version = get_short(t);
  ip->createDate = pilot_time_to_unix_time (get_long(t));
  ip->modifyDate = pilot_time_to_unix_time (get_long(t));
  ip->backupDate = pilot_time_to_unix_time (get_long(t));
  ip->modnum = get_long(t);
  app_info_offset = get_long(t);
  checkoffset(&db, app_info_offset);
  ip->appInfo = text(app_info_offset ? t->chars + app_info_offset : NULL, 0);
  sort_info_offset = get_long(t);
  checkoffset(&db, sort_info_offset);
  ip->sortInfo = text(sort_info_offset ? t->chars + sort_info_offset : NULL, 0);
  get_chars(ip->type, t, 4);
  ip->type[4] = 0;
  get_chars(ip->creator, t, 4);
  ip->creator[4] = 0;
  ip->uidSeed = get_long(t);
  ip->nextRecordListID = get_long(t);
  ip->numrecs = get_short(t);
  
  assert(chars_used(t) == 78);

  if (ip->nextRecordListID != 0)
    fprintf (stderr, "Warning -- nonzero next record list ID.\n");
        
  assert(!ip->resource);
  assert((short)ip->numrecs >= 0);

  db.records = malloc(ip->numrecs * sizeof(*db.records));
  assert(db.records);

  for (i = 0; i < ip->numrecs; i++) {
    unsigned long offset = get_long(t);
    unsigned long n;
    checkoffset(&db, offset);
    db.records[i].data = text(db.image.chars + offset, 0);
    n = get_long(t);
    db.records[i].uid = n & 0xffffff;
    if (i > 0) {
      db.records[i-1].data.len = db.records[i].data.chars - db.records[i-1].data.chars;
      assert(db.records[i-1].data.len >= 0);
    }
    flags = (n >> 24) & 0xff;
    db.records[i].secret   = flags & Rsecret ? 1 : 0;
    db.records[i].busy     = flags & Rbusy   ? 1 : 0;
    db.records[i].dirty    = flags & Rdirty  ? 1 : 0;
    db.records[i].delete   = flags & Rdelete ? 1 : 0;
    db.records[i].category = flags & 0xf;
  }
  if (i > 0) {
    db.records[i-1].data.len = 
      db.image.chars + db.image.len - db.records[i-1].data.chars;
    assert(db.records[i-1].data.len >= 0);
  }

  db.filler = get_short(t);

  t->p = (unsigned char *)t->chars; /* done reading; reset */

  { char *p = ip->numrecs > 0 ? db.records[0].data.chars : t->chars + t->len;
    if (ip->sortInfo.chars) {
      ip->sortInfo.len = p - ip->sortInfo.chars;
      assert(ip->sortInfo.len >= 0);
      p = ip->sortInfo.chars;
    }
    if (ip->appInfo.chars) {
      ip->appInfo.len = p - ip->appInfo.chars;
      assert(ip->appInfo.len >= 0);
      p = ip->appInfo.chars;
    }
  }
  return db;
}

Defines read_db (links are to index).

<functions>+= (<-U) [<-D->]
extern int write_dbinfo (FILE *fp, struct DBInfo *ip, unsigned long aoff,
                         unsigned long soff) {
  unsigned short flags;
  long pos = ftell(fp);

  put_chars(fp, ip->name, 32);
  ip->name[32] = 0;
  flags = 0;
  if(ip->resource) flags |= Fresource;
  if(ip->dirtyAppInfo) flags |= FdirtyAppInfo;
  if(ip->backup) flags |= Fbackup;
  if(ip->installOver) flags |= FinstallOver;
  if(ip->reset) flags |= Freset;
  if(ip->readonly) flags |= Freadonly;
  put_short(fp, flags);
  put_short(fp, ip->version);
  put_long(fp, unix_time_to_pilot_time(ip->createDate));
  put_long(fp, unix_time_to_pilot_time(ip->modifyDate));
  put_long(fp, unix_time_to_pilot_time(ip->backupDate));
  put_long(fp, ip->modnum);
  put_long(fp, aoff);
  put_long(fp, soff);
  put_chars(fp, ip->type, 4);
  put_chars(fp, ip->creator, 4);
  put_long(fp, ip->uidSeed);
  put_long(fp, ip->nextRecordListID);
  put_short(fp, ip->numrecs);
  assert(ftell(fp) - pos == 78);
  return 78;
}

<functions>+= (<-U) [<-D->]
void write_record_header(FILE *fp, struct record *rec, unsigned offset) {
  unsigned long flags;
  put_long(fp, offset);
  
  flags = 0;
  if (rec->secret  ) flags |= Rsecret;
  if (rec->busy    ) flags |= Rbusy  ;
  if (rec->dirty   ) flags |= Rdirty ;
  if (rec->delete  ) flags |= Rdelete;
  flags |= rec->category & 0xf;
  put_long(fp, (flags << 24) | rec->uid);
}

Defines write_record_header (links are to index).

<functions>+= (<-U) [<-D->]
void unpack_categories(FILE *out, struct PDBCategoryAppInfo *cat, int cols) {
  int i;
  fprintf(out, "\n        { lastuid = %d, ", cat->lastUniqueID);
  fprintf(out, "\n          categories = {");
  for (i = 0; i < 16; i++) {
    pushchar(&(cat->name[i][16]));
    if (i % cols == 0) fprintf(out, "\n            ");
    fprintf(out, "{ name = %s, renamed = %s, uid = %d }%s",
            quote(cat->name[i], 1), cat->renamed[i] ? "1" : "nil", cat->ID[i],
            comma(i + 1 < 16));
    popchar();
  }
  fprintf(out, "\n          }");
  fprintf(out, "\n        }");
}

Defines unpack_categories (links are to index).

<functions>+= (<-U) [<-D->]
struct PDBCategoryAppInfo pack_categories(void) {
  struct PDBCategoryAppInfo cat;
  int i;
  if (lua_dostring("if not istable(Database.appInfo.categories) then "
        "lua_error('Database application info does not have categories') end"))
    assert(0);
  cat.lastUniqueID = intval("Database.appInfo.categories.lastuid");
  for (i = 0; i < 16; i++) {
    char buf[90];
    sprintf(buf, "thiscat = Database.appInfo.categories.categories[%d]", i+1);
    if (lua_dostring(buf)) assert(0);
    setstringopt(cat.name[i], "thiscat.name", "", 16);
    cat.renamed[i] = bitval("thiscat.renamed");
    cat.ID[i] = intval("thiscat.uid");
  }
  return cat;
}

Defines pack_categories (links are to index).

<functions>+= (<-U) [<-D->]
struct PDBCategoryAppInfo catread(Text *t) {
  struct PDBCategoryAppInfo ai;
  unsigned short r;
  int i;
  assert (t->p);
  r = get_short(t);
  for (i=0;i<16;i++)
    ai.renamed[i] = (r & (1<<i)) ? 1 : 0;
  for(i=0;i<16;i++)
    get_chars(ai.name[i], t, 16);
  get_chars(ai.ID, t, 16);
  ai.lastUniqueID = get_byte(t);
  return ai;
}

Defines catread (links are to index).

<functions>+= (<-U) [<-D]
int catwrite(FILE *fp, struct PDBCategoryAppInfo *ai) {
  int n = 2 + 16*16 + 16 + 1;
  if (fp) {
    unsigned short r = 0;
    long pos = ftell(fp);
    int i;
    for (i=0;i<16;i++)
      if (ai->renamed[i]) r |= 1<<i;
    put_short(fp, r);
    for(i=0;i<16;i++)
      put_chars(fp, ai->name[i], 16);
    put_chars(fp, ai->ID, 16);
    put_byte(fp, ai->lastUniqueID);
    assert (ftell(fp) - pos == n);
  }
  return n;
}

Defines catwrite (links are to index).

<*>: D1
<functions>: U1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11
<header>: D1, D2, D3, D4, D5, D6, U7

catread: U1, D2
catwrite: U1, D2
checkoffset: D1, U2
DB: U1, D2
pack_categories: U1, D2
PILOT_TIME_DELTA: D1
read_db: U1, D2
Record: D1
time_t: U1, D2
unix_time_to_pilot_time: D1, U2
unpack_categories: U1, D2
write_record_header: U1, D2