Phicore data file format¶

Version: 2.0, May 25 2018, LIDYL CEA

This document describes the API of the file format used to store spatio-temporal laser beam profiles.

I. HDF5 file format¶

Overview¶

Hierarchical Data Format (HDF5) is designed to efficiently store large amounts of scientific data. A phicore data file, is an HDF5 file with the following structure,

|-metadata
--data/
  |_ variable1  int16, shape=(Nx, Ny, Nw)
  |_ variable2  float32, shape=(Nx, Ny, Nt)
--scales/
  |_ variable1_x float64, shape=(Nx,)
  |_ variable1_y float64, shape=(Ny,)
  |_ variable1_w float64, shape=(Nw,)
  |_ variable2_x float64, shape=(Nx)
  |_ variable2_y float64, shape=(Ny)
  |_ variable2_t float64, shape=(Nt)
   ...
--diag/
  |_ reference_image1
  |_ reference_image2

Thus, phicore files always have at least the following three groups,

data : a group with N-dimensional variables (in this case variable1 and variable2)
scales : contains 1D vectors with axis coordinates (time, space, etc) for all variables. in the above example, variable1 is 3D and has [x, y, w] coordinates.
diag : a group with custom variables. For instance it could contain reference images etc.

In addition, phicore files store metadata as specified below.

Data variables¶

Data variables in the data/ group can be 2D or 3D array of integer or float data type.

Spatial dimensions are the first ones, while the temporal or frequency dimension comes last.

To save disk usage, individual variables can be stored with compression filters including zlib, szip as well as blosc and lz4.

Metadata¶

Root metadata

The root HDF5 node node must contain at least the following metadata attributes,

rev_fileformat (_float_): File format revision. Should be equal to 3.

date (_string_): “YYYY-MM-DD-HHMMSS”

as well as the following optional attributes,

operator (_string_): Author name

comments (_string_): Free format field.

data_source (_string_): “simulation” OR “experiment”

other attributes can be added as needed.

Node metadata¶

Each data variable must store the following metadata attributes,

name (_string_): the variable name

scales (_string_): a list of data coordinates (e.g. ['x', 'y', 'w'])

as well as the following optional attributes,

data_source (_string_): “simulation” OR “experiment”

Each scale/coordinate variable must include the following attributes,

unit (_string_): a string representation of coordinate unit

Physical quantities¶

Data variable notations

The variable names are Intensity (I), Amplitude (A), Complex electric field (E) and Phase (ph).

Coordinate notations

Description	Name	Units
Spatial coordinates	`x` or `y`	mm
Spatial frequencies	`kx` or `ky`	1/mm
Time	`t`	fs
Optical delay	`tau`	fs
Optical path difference	`d`	nm
Angular frequency	`w`	PHz.rad
Frequency	`f`	PHz
Wavelength	`lamb`	nm

Visualization software¶

Visualization software for the phicore data format is developed at LIDYL, CEA.

Raw data in the HDF5 can be displayed by a variety of HDF5 viewers, including HDF Compass, HDF view and ViTabes.

II. Client libraries¶

HDF5 supports client libraries in numerous languages, including Java, MATLAB, Scilab, Octave, Mathematica, IDL, Python, R, and Julia however the phicore data format specification for each of these still needs to be implemented.

In the following, we will focus on the Python language.

Python¶

Overview A client library for the phicore data format is implemented at LIDYL, CEA and is available upon request.

The general approach consists in mapping the HDF5 file structure to labeled N-dimensional array representation provided by the xarray package.

Each data variable can be represented as an xarray.DataArray, while the set of all data variables corresponds to xarray.Dataset.

Specification

Each data variable X must be a valid 2D or 3D xarray object (see xarray API) and include the following attributes,

scale_units (dict): a dictionary mapping dimensions names (X.dims) to their units types (str).

As a result of the “Physical quantities” section, X.dim must be of the form [x|kx, y|ky, lamb|w|f|tau|d], while X.coords must be a dict-like object with these same keys.

Example

A data variable X represented as an xarray.DataArray can be saved to disk with,