This is tdl (Tiny Data Language)

Tdl is a data processing language written in python and designed to be

• easy to use for novices.
• complete enough for intermediate to advanced data processing.
• data-centric, so that arrays of data are easy to manage and use.
• easily extensible with python.

Tdl uses numpy (version 0.9.6 or higher) for all its numerical functionality, so processing arrays of data is easy. Tdl will also soon require scipy, as this will allow 'advanced numeric capabilities', including non-linear least squares minimization and ffts.

Features

Variables

As with other programming languages, variables are named quantities used to hold data. In tdl, a variable may contain any of the following data types: integers, real numbers, complex numbers, strings, lists, dictionaries (hashes), and multidimensional arrays of numerical data. Tdl does not associate a variable's type with its name, and does not require the type of a variable to be specified. A variable name must start with a letter or '_', followed any number of letters, numbers, and '_'. Variable names are case-sensitive.

To assign a value to a variable, you would say:

    tdl> x = sqrt(90) -2.
    tdl> print x
    7.48683298051

Where the '=' sign indicates the assignment, and the value of the variable of the left-hand-side of the '=' sign is set to the value of the expression on the right-hand-side of the '='.

Once they exist, variables can be used in expressions:

    tdl> x_squared = x * x
    tdl> print x_squared 
    56.052668078

Array data (that is, sequences of numbers) and strings (sequences of characters) can be accessed either element-by-element or with slices:

      tdl> arr = arange(10)    # simple array 
      tdl> print arr 
      [0 1 2 3 4 5 6 7 8 9 10]
      tdl> print arr[3]
      3
      tdl> print arr[2:8]
      [2 3 4 5 6 7]

      tdl> str = 'here is a string'
      tdl> print str[0:6]
      here i

the underlying data types are all Python (or numpy) types.

Groups and Namespaces

Data often comes with many related quantities, typically with different types. In tdl, variables can be (and indeed, are) grouped together into sets of variables called groups. Besides having a name, a group is really just a convenient way to associate different variables with one another.

But since each group has a name, each variable really has 2 (or more) names: a variable name and one (or more) group names. The different parts of the name are separated by a '.', so that a full name for a variable might be 'group1.energy' or 'dataset1.x'. Groups can also be nested, giving names like 'group.subgroup.subsubgroup.value'.

As in the examples above, it is not always necessary to refer to variables by their full name. Unqualified names (ie, those without a 'group.' prefix) will be searched for in a list of groups. In addition, there is always a "Local Group" which is the first group looked in, and where new variables will be placed. At startup, the Local Group is '_main', but this can be changed, as we'll see below.

     tdl> x = 1
     tdl> group1.x = 3.3
     tdl> print x, _main.x, group1.x
     1 1 3.3

In addition to _main, there are a few other groups at startup: _builtin

• holds constants and builtin functions, and _sys holds program state information.

This _builtin group is special in that it cannot be easily altered from tdl directly. There are also groups _math which holds many mathematical operators.

complete and clean syntax for programming

• The syntax is python-like, with some exceptions. Most importantly, indentation doesn't matter, and blocks end with 'end' statements. A typical for-block will look like this:

       for i in range(10):
           print i
       endfor     

• there are also while blocks and if-elif-else blocks:

       n = 0
       while n<10: 
          print ' n = ',n
          if n==3: print 'here is 3!!'
          n = n + 1
       endwhile
     
       if n > 10:
          print 'No'
       elif n > 5 and n < 8:
          print 'Maybe'
       else:
          print 'Yep'
       endif

• A design goal is that well-formed tdl code should be very easy to translate into valid python (and vice versa).

user-defined functions (aka procedures)

• User defined functions can be written in tdl:

        def  myfunc(arg1, option='test'):
            'documentation string'
            print 'this is my funcition ', arg1
            print 'optional param = ', option
            if type(option) != 'string':     
                 print 'option must be a string!!'
                return False
            endif
            value = sqrt(arg1)
            x.tmp = value
            return value > 10.
        enddef

• which could be called as

        tdl> ret = myfunc(3., option = 'xx')
        tdl> print ret, x.tmp
        False 1.73205080757

Other features:

• load() function:
  • you can run a file of tdl commands with the load() function:

       tdl> load('myfile.tdl')

• eval() function:
  • you can construct a tdl expression on the fly from strings and execute it:

       tdl> eval("%s.value = %f" %  ('group', 10.2))
       tdl> print group.value 
       10.2

• read_ascii() function:
  • you can read in ASCII column data files very easily.

      tdl> read_ascii('my.dat', group='f')
      ['f', 'x', ''y']

• this read 2 columns (labeled 'x' and 'y') from the column file and created the array variables f.x and f.y. Also created was

       f.titles 

• to hold the titles in the data file (non-numeric content at the top of the file) and

       f.column_labels

• on-line help: still in progress.
• easy to add your python functions, including getting access to all
  • the 'data groups' inside your python function.
    • ...
    see tdl_read_ascii in tdl/TdlLib.py

Differences between tdl and Python

* tdl has many builtins and assumes numerical data (that is, arrays of numbers). * tdl has no tuples. Lists are used in their place. * indentation does not matter. blocks are ended with 'end***'

• (if / endif , fo/endfor, def/enddef)

* when in doubt, assignment makes a copy, and does not give a reference * data types are not generally objects, and have no methods associated

• with them. Tdl use procedural approaches:

     x = arange(100)
     reshape(x,[10,10])   instead of x.shape = (10,10)