(Demeter main page) (Cookbook main page)

Data Handling

This section covers the parts of Demeter that implement the functionality of Athena.

Contents

Data Handling
1. Importing data
2. Plotting and processing data

Importing data

How do I import a mu(E) data file and make a plot?

In this recipe, we import one of the iron foil examples that comes in the Demeter package. This file has already been converted from raw data into mu(E) data. Demeter will recognize that it is mu(E) data and process it accordingly.

   1 #!/usr/bin/perl   ### import and plot
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 my $data = Demeter::Data -> new();
   6 $data -> set(file        => "data/fe.060.xmu",
   7              name        => 'Fe 60K',
   8              bkg_rbkg    => 1.5,
   9              bkg_spl1    => 0,    bkg_spl2    => 18,
  10              bkg_nor1    => 100,  bkg_nor2    => 1800,
  11              fft_kmax    => 3,    fft_kmin    => 17,
  12             );
  13 $data -> plot('E');
  14 
  15 sleep 2;
  16 $data -> po -> start_plot;
  17 $data -> plot('k');
  18 
  19 sleep 2;
  20 $data -> po -> start_plot;
  21 $data -> plot('R');

After reading the data and setting several of the data processing parameters, the data are plotted in energy, then as chi(k), then as chi(R), with a two second pause in between plots. The calls to start_plot at lines 16 and 20 reinitialize the plotting system so that the subsequent plot is a new plot rather than an overplot.

Note that you do not need to explicitly do a background removal or a Fourier transform. In fact, it should never be necessary to do so. Just make a plot and let Demeter keep track of what needs to be done in order to make that plot.

How do I import raw data from the beamline?

To convert raw data into mu(E) data you need to know which columns contain which scalars from the measurement. In this example of transmission data, column 1 is the energy, column 2 is I0, and column 3 is transmission.

   1 #!/usr/bin/perl    ### import raw data
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 my $data = Demeter::Data -> new();
   6 $data -> set(file        => "data/fe.060.xmu",
   7              name        => 'Fe 60K',
   8              energy      => '$1', # column 1 is energy
   9              numerator   => '$2', # column 2 is I0
  10              denominator => '$3', # column 3 is It
  11              ln          => 1,    # these are transmission data
  12              bkg_pre1    => -30,  bkg_pre2    => -150,
  13              bkg_nor1    => 150,  bkg_nor2    => 1757.5,
  14              bkg_spl1    => 0.5,  bkg_spl2    => 22,
  15              fft_kmax    => 3,    fft_kmin    => 14,
  16             );
  17 $data -> set(file => "data/fe.060", label => 'Iron foil',);
  18 $data -> plot('E');

The energy, numerator, and denominator must be explicitly specified, as is a flag for whether the natural log should be taken. The "language" of the column selection was chosen to resemble gnuplot's syntax. Note that you must quote those strings containing dollar signs with single quotes. If you use double quotes, you will need to escape the dollar signs like so: "\$1".

Fluorescence data would have the ln flag set to zero and the I0 column would be in the denominator. For multi-element data, you might have a sum of columns for the numerator -- something like '$5+$6+$7+$8'.

How do I import data from an Athena project file?

There is a Demeter object specifically intended for this purpose.

   1 #!/usr/bin/perl   ### import and plot
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 
   6 my $prj = Demeter::Data::Prj -> new(file=>'iron_data.prj');
   7 
   8 my $data = $prj -> record(1);
   9 $data -> plot('E');
  10 ## and so on...

At line 6, the Athena project file is parsed and a list of available data groups is generated. At line 8, the first record from that Athena project file is imported into a Data object. The various processing parameters are also imported from the project file, thus the call to the set method is skipped in this example. This $data, imported via the Prj object, is a normal Demeter Data object.

The lsprj program that comes with the Demeter distribution makes use of the Prj object.

Plotting and processing data

How do I plot multiple data sets?

Answered in the simplest possible terms, define more than one Data object and plot them all. Here is the simplest approach:

   1 #!/usr/bin/perl     ### plot more than one data set
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 my $d1 = Demeter::Data -> new();
   6 $d1 -> set(file        => "data/fe.060.xmu",
   7            name        => 'Fe 60K',
   8            bkg_rbkg    => 1.5,
   9            bkg_spl1    => 0,    bkg_spl2    => 18,
  10            bkg_nor1    => 100,  bkg_nor2    => 1800,
  11            fft_kmax    => 3,    fft_kmin    => 17,
  12           );
  13 $d1 -> plot('E');
  14 my $d2 = Demeter::Data -> new();
  15 $d2 -> set(file        => "data/fe.300.xmu",
  16            name        => 'Fe 300K',
  17            bkg_rbkg    => 1.5,
  18            bkg_spl1    => 0,    bkg_spl2    => 18,
  19            bkg_nor1    => 100,  bkg_nor2    => 1800,
  20            fft_kmax    => 3,    fft_kmin    => 17,
  21           );
  22 $d2 -> plot('E');

This has a big problem -- it is very repetitious. Repetition is tedious and error-prone. There are two solutions to this problem:

Use named arrays

The first is to recognize that the argument of the set method is just an array. That means we can use a named array to specify the parameters the two data sets have in common:

   1 #!/usr/bin/perl          ### defining attributes with an anonymous hash
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 my @common_attributes = (bkg_rbkg    => 1.5,
   6                          bkg_spl1    => 0,    bkg_spl2    => 18,
   7                          bkg_nor1    => 100,  bkg_nor2    => 1800,
   8                          fft_kmax    => 3,    fft_kmin    => 17,
   9                         );
  10 
  11 my $d1 = Demeter::Data -> new();
  12 $d1 -> set(file => "data/fe.060.xmu", name => 'Fe 60K');
  13 $d1 -> set(@common_attributes);
  14 
  15 my $d2 = Demeter::Data -> new();
  16 $d2 -> set(file => "data/fe.300.xmu", name => 'Fe 300K');
  17 $d2 -> set(@common_attributes);
  18 
  19 foreach my $d ($d1, $d2) {
  20   $d -> plot('E');
  21 };

Clone objects

The other shortcut is to use the clone method:

   1 #!/usr/bin/perl     ### cloning data groups
   2 ## -----------------------------------------------------------
   3 use Demeter; ## strict and warnings imported automatically with Demeter
   4 ## -----------------------------------------------------------
   5 my $d1 = Demeter::Data -> new();
   6 $d1 -> set(file        => "data/fe.060.xmu",
   7            name        => 'Fe 60K',
   8            bkg_rbkg    => 1.5,
   9            bkg_spl1    => 0,    bkg_spl2    => 18,
  10            bkg_nor1    => 100,  bkg_nor2    => 1800,
  11            fft_kmax    => 3,    fft_kmin    => 17,
  12           );
  13 my $d2 = $d1 -> clone(file => "data/fe.300.xmu", label => 'Fe 300K');
  14 
  15 foreach my $d ($d1, $d2) {
  16   $d -> plot('E');
  17 };

The cloning at line 15 returns a replica of the original data, but with the specified attributes changed. Since the file attribute is changed, the reading of the new data is triggered -- thus the clone is not a strict copy (in the sense of Athena's "copy group" function). All attributes are cloned and associated with the new data.

There is no advantage to one of these approaches over the other and they can be used interchangeably. Take your pick!

How do I align and merge data?

Once you have imported data and set the data processing parameters using one of the recipes above, you can align data sets. AS in Athena, merging data sets requires that you first align them.

Alignment

This script from the Demeter distribution demonstrates data alignment. The salient lines are
```
   1 $d0->align($d1);
   2 $d1->e0($d0);
```
These lines align data group $d1 to $d0 and sets the e0 value of $d1 to be the same as $d0.

Merging

This script from the Demeter distribution demonstrates data merging. The salient line is
```
   1 my $merge = $d0->merge("N", $d1, $d2);
```
which tells Demeter to merge Data objects $d1 and $d2 with $d0 after interpolating $d1 and $d2 onto the grid of $d0. The first argument to the merge method tells which form of the data to do the merge in. In this case "N" is for "normalized". The other options are "X" and "K" for merging raw mu(E) data or chi(k) data. Note that $merge is a Data object containing the merged data.

How do I <X> data? [ where <X> = (calibrate|truncate|deglitch|smooth|add noise to|convolve) ]

Once you have imported data and set processing parameters using one of the recipes above, you can perform a number of data processing chores. A script demonstrating these comes with Demeter. See examples/harness or look here.

calibrate

Calibration is the process of setting a particular point in the data set to a particular value.
```
   1 $d0->calibrate($selected_energy, $edge_energy);
```
The two arguments are optional. If not provided, an energy very close to the first big peak in the first derivative of mu(E) will be used as the selected energy and the tabulated value will be used as the edge energy.

truncate

This simply removes data before or after a selected energy value:
```
   1 $d0 -> Truncate('before', 7100);
```
or
```
   1 $d0 -> Truncate('after', 7600);
```
Example script.

deglitch

Individual points can be removed from a data set like so:
```
   1 $d0 -> deglitch(17385.686, 17655.5);
```
Demeter does not, at this time, provide a specific manner of obtaining the energy values of the glitches. The cursor method (see the documentation for Demeter::Dispose) provides a way of plucking values interactively from the PGPLOT window. Note that Athena's algorithm for finding glitches en masse using margins around the normalization lines has not yet been implemented.BR[http://cars9.uchicago.edu/svn/demeter/trunk/examples/tests/deglitch.pl Example script.]

smooth

Ifeffit's three-point smoothing algorithm can be applied repeatedly:
```
   1 $d0 -> smooth($n);
```
Here $n is the number of repititions. BR[http://cars9.uchicago.edu/svn/demeter/trunk/examples/tests/smooth.pl Example script.]

add noise

Noise can be added to mu(E) or chi(k) data. For mu(E) data, the noise parameter is a fraction of the edge step. For chi(k) data, the noise parameter is referenced to the size of the chi(k) data. Note that, unlike most other data processing methods, this method takes a hash reference as its argument.
```
   1 $d1 -> noise(noise=>0.02, which=>'xmu');
```
Example script.

convolve

Data can be convoluted with with either a Gaussian or a Lorentzian as mu(E) or chi(k). Note that, unlike most other data processing methods, this method takes a hash reference as its argument.
```
   1 $d1 -> convolve(width=>2, type=>'Gaussian', which=>'xmu');
```
The width is measured in eV, the type can be either Gaussian or Lorentzian , which can be either xmu or chi.BR[http://cars9.uchicago.edu/svn/demeter/trunk/examples/tests/convolve.pl Example script.]