Output formatting

Caliper provides a variety of formatters to create machine or human-readable output. These can be used in all I/O services and tools that process CalQL, e.g. cali-query, the report and mpireport services, as well as the report generation API.

Formatter configuration

To select a formatter, use the FORMAT statement in CalQL. Some formatters, e.g. tree, take additional arguments in a “function call” syntax, e.g.

FORMAT tree(source.function#callpath.address)

The SELECT CalQL statement determines the attributes that will be written by the formatter (all other attributes will be ignored). If applicable, attributes will be printed in the order they are listed in the SELECT statement.

Some formatters (e.g., table) support the ORDER BY CalQL statement to print records in a given order, but not all of them do.

Cali

The cali formatter writes Caliper raw data that can be read by cali-query, the Python reader API, and the caliper_native_reader importer in Hatchet.

Expand

The expand formatter writes records as comma-separated key-value lists, with each record in a single line. Hierachical values are written with slash / separators.

Example:

SELECT event.end#function,count(),sum(time.inclusive.duration) GROUP BY event.end#function FORMAT expand

Output:

event.end#function=TimeIncrement,count=100,time.inclusive.duration=1280
event.end#function=IntegrateStressForElems,count=100,time.inclusive.duration=358512
event.end#function=CalcFBHourglassForceForElems,count=100,time.inclusive.duration=700886
event.end#function=CalcHourglassControlForElems,count=100,time.inclusive.duration=1073630
event.end#function=CalcVolumeForceForElems,count=100,time.inclusive.duration=1456613
event.end#function=CalcForceForNodes,count=100,time.inclusive.duration=1466051
event.end#function=LagrangeNodal,count=100,time.inclusive.duration=1508828

Table

The table formatter prints a human-readable text table with automatically determined column widths. It supports ORDER BY.

Example:

SELECT event.end#function,count(),sum(time.inclusive.duration) GROUP BY event.end#function FORMAT table ORDER BY time.inclusive.duration DESC

Output:

event.end#function              count time.inclusive.duration
main                                1                 3395643
LagrangeLeapFrog                  100                 3379753
LagrangeElements                  100                 1755059
LagrangeNodal                     100                 1508828
CalcForceForNodes                 100                 1466051
CalcVolumeForceForElems           100                 1456613
ApplyMaterialPropertiesForElems   100                 1163596
EvalEOSForElems                  1100                 1113417
CalcHourglassControlForElems      100                 1073630
CalcEnergyForElems               3500                  805605
CalcFBHourglassForceForElems      100                  700886
IntegrateStressForElems           100                  358512
CalcPressureForElems            10500                  324182
CalcQForElems                     100                  309658
CalcLagrangeElements              100                  266492
CalcKinematicsForElems            100                  241132
CalcMonotonicQGradientsForElems   100                  149492
CalcTimeConstraintsForElems       100                  110236
CalcMonotonicQRegionForElems     1100                   98944
CalcCourantConstraintForElems    1100                   49281
CalcHydroConstraintForElems      1100                   33125
CalcSoundSpeedForElems           1100                   27975
UpdateVolumesForElems             100                    8147
TimeIncrement                     100                    1280

Tree

The tree formatter prints records in hierarchical form based on the input data, such as nested functions and regions.

The attribute used to build the hierarchy can be given as an argument to the formatter (e.g. FORMAT tree(function) to use function). By default, the hierarchy is built from all attributes flagged as NESTED, which includes the default function, loop, and region annotations.

Example:

SELECT count(),sum(time.inclusive.duration) WHERE event.end#function GROUP BY function FORMAT tree

Output:

Path                                       count time.inclusive.duration
main                                           1                 3395643
  LagrangeLeapFrog                           100                 3379753
    CalcTimeConstraintsForElems              100                  110236
      CalcHydroConstraintForElems           1100                   33125
      CalcCourantConstraintForElems         1100                   49281
    LagrangeElements                         100                 1755059
      UpdateVolumesForElems                  100                    8147
      ApplyMaterialPropertiesForElems        100                 1163596
        EvalEOSForElems                     1100                 1113417
          CalcSoundSpeedForElems            1100                   27975
          CalcEnergyForElems                3500                  805605
            CalcPressureForElems           10500                  324182
      CalcQForElems                          100                  309658
        CalcMonotonicQRegionForElems        1100                   98944
        CalcMonotonicQGradientsForElems      100                  149492
      CalcLagrangeElements                   100                  266492
        CalcKinematicsForElems               100                  241132
    LagrangeNodal                            100                 1508828
      CalcForceForNodes                      100                 1466051
        CalcVolumeForceForElems              100                 1456613
          CalcHourglassControlForElems       100                 1073630
            CalcFBHourglassForceForElems     100                  700886
          IntegrateStressForElems            100                  358512
  TimeIncrement                              100                    1280

Json

The json formatter writes a list-of-dicts style json file, where each Caliper record is a JSON object with attribute:value entries. Hierarchical values are written as a single string with a slash (/) as separator. Note that the json formatter usually produces larger files than json-split.

Optional arguments:

pretty

More human-readable output (inserts extra whitespace and line breaks).

split

Just write the records, skip the outer “[ ]” array delimiters.

quote-all

Write all values as strings using " quotes, even numerical ones.

Example:

SELECT function,loop,count(),sum(time.inclusive.duration) WHERE event.end#function GROUP BY function,loop FORMAT json

Output:

[
{"function":"main","count":1,"time.inclusive.duration":3395643},
{"loop":"lulesh.cycle","function":"main/TimeIncrement","count":100,"time.inclusive.duration":1280},
{"loop":"lulesh.cycle","function":"main/LagrangeLeapFrog","count":100,"time.inclusive.duration":3379753},
{"loop":"lulesh.cycle","function":"main/LagrangeLeapFrog/LagrangeNodal","count":100,"time.inclusive.duration":1508828},
{"loop":"lulesh.cycle","function":"main/LagrangeLeapFrog/LagrangeNodal/CalcForceForNodes","count":100,"time.inclusive.duration":1466051},
{"loop":"lulesh.cycle","function":"main/LagrangeLeapFrog/LagrangeNodal/CalcForceForNodes/CalcVolumeForceForElems","count":100,"time.inclusive.duration":1456613},

Json-split

The json-split formatter writes a JSON file with separate fields for Caliper records and metadata. It is generally a more efficient format than the list-of-dicts produced by the json formatter, and explicitly retains hierarchical information. Attributes flagged as NESTED (e.g., default annotations such as annotation, function, and loop) will be merged into the path column to retain hierarchical information between them.

The top-level object contains the following fields:

data

The Caliper records as a 2D array (array of arrays). Each row is a record, each column is an attribute. The attribute/column labels are stored in the columns array. Missing values in a record are written as null entries.

A column is either a value or reference entry. Value entries are stored directly in each record, whereas reference entries contain the index of the corresponding generalized context tree node in the nodes array. The is_value entry in the column_metadata array determines which columns are value and which are reference entries.

columns

The labels (i.e., attribute names) of the data columns as array of strings.

column_metadata

Any metadata for the data columns as array of objects. The objects contain the is_value entry, which determines if the corresponding data column stores a value or a reference entry.

nodes

An array of metadata node objects. A reference entry in a data record refers to the node at the corresponding index in this array.

The metadata nodes generally form a tree (or forest), retaining hierarchical information. Nodes have a label entry with the nodes’ value, and an optional parent entry, which is the index of the node’s hierarchical parent node in the nodes array. It is guaranteed that a parent node is placed before all of its children in the array.

Example:

SELECT function,loop,count(),sum(time.inclusive.duration) GROUP BY function,loop FORMAT json-split

Output:

{
   "data": [
    [ 1, 3395643, 0 ],
    [ 100, 1280, 2 ],
    [ 100, 3379753, 3 ],
    [ 100, 1508828, 4 ],
    [ 100, 1466051, 5 ],
    [ 100, 1456613, 6 ],
    [ 100, 358512, 7 ],
    [ 100, 1073630, 8 ],
    [ 100, 700886, 9 ],
    [ 100, 1755059, 10 ],
    [ 100, 266492, 11 ],
    [ 100, 241132, 12 ],
    [ 100, 309658, 13 ],
    [ 100, 149492, 14 ],
    [ 1100, 98944, 15 ],
    [ 100, 1163596, 16 ],
    [ 1100, 1113417, 17 ],
    [ 3500, 805605, 18 ],
    [ 10500, 324182, 19 ],
    [ 1100, 27975, 20 ],
    [ 100, 8147, 21 ],
    [ 100, 110236, 22 ],
    [ 1100, 49281, 23 ],
    [ 1100, 33125, 24 ]
  ],
  "columns": [ "count", "time.inclusive.duration", "path" ],
  "column_metadata": [ { "is_value": true }, { "is_value": true }, { "is_value": false }  ],
  "nodes": [ { "label": "main" }, { "label": "lulesh.cycle", "parent": 0 }, { "label": "TimeIncrement", "parent": 1 }, { "label": "LagrangeLeapFrog", "parent": 1 }, { "label": "LagrangeNodal", "parent": 3 }, { "label": "CalcForceForNodes", "parent": 4 }, { "label": "CalcVolumeForceForElems", "parent": 5 }, { "label": "IntegrateStressForElems", "parent": 6 }, { "label": "CalcHourglassControlForElems", "parent": 6 }, { "label": "CalcFBHourglassForceForElems", "parent": 8 }, { "label": "LagrangeElements", "parent": 3 }, { "label": "CalcLagrangeElements", "parent": 10 }, { "label": "CalcKinematicsForElems", "parent": 11 }, { "label": "CalcQForElems", "parent": 10 }, { "label": "CalcMonotonicQGradientsForElems", "parent": 13 }, { "label": "CalcMonotonicQRegionForElems", "parent": 13 }, { "label": "ApplyMaterialPropertiesForElems", "parent": 10 }, { "label": "EvalEOSForElems", "parent": 16 }, { "label": "CalcEnergyForElems", "parent": 17 }, { "label": "CalcPressureForElems", "parent": 18 }, { "label": "CalcSoundSpeedForElems", "parent": 17 }, { "label": "UpdateVolumesForElems", "parent": 10 }, { "label": "CalcTimeConstraintsForElems", "parent": 3 }, { "label": "CalcCourantConstraintForElems", "parent": 22 }, { "label": "CalcHydroConstraintForElems", "parent": 22 } ]
}