# JLS Format ### *group* File format JLS file format. JLS file format. ### Defines ### JLS_FORMAT_VERSION_MAJOR ### JLS_FORMAT_VERSION_MINOR ### JLS_FORMAT_VERSION_PATCH ### JLS_FORMAT_VERSION_U32 ### JLS_HEADER_IDENTIFICATION The file identification bytes at the start of the file. These bytes are not arbitrary. We carefully selected them to provide: * Identification: Help the application determine that this file is in the correct format with minimal uncertainty. * Correct endianness: Little endian has won, so this entire format is stored in little endian format. * Proper binary processing: The different line ending combinations ensure that the reader is not “fixing” the line endings, since this is a binary file format. * Display: Include “substitute” and “file separator” so that text printers to not show the rest of the file. The following table describes maps each byte to its purpose: | Value (hex) | Purpose | |-------------------|-----------------------------------------------------| | 6A 6C 73 66 6d 74 | ASCII “jlsfmt”, when viewed in text editor. | | 0D 0A | DOS line ending to ensure binary correctness. | | 20 | ASCII space. | | 0A | UNIX line ending to ensure binary correctness. | | 20 | ASCII space. | | 1A | Substitute character (stops listing under Windows). | | 20 20 | ASCII spaces. | | B2 | Ensure that system supports 8-bit data | | 1C | File separator. | ### JLS_SOURCE_COUNT The maximum allowed number of sources. ### JLS_SIGNAL_COUNT The maximum allowed number of signals. ### JLS_SUMMARY_LEVEL_COUNT The number of summary levels. ### JLS_TRACK_TAG_FLAG ### JLS_TRACK_TAG_PACK(track_type, track_chunk) Pack the chunk tag. * **Parameters:** * **track_type** – The [jls_track_type_e](#group__jls__format_1ga5b64213e64bc8a8ddfd7277f6230b153) * **track_chunk** – The [jls_track_chunk_e](#group__jls__format_1gad59db733d67135892ad5b8525aceaf2d) * **Returns:** The tag value. ### JLS_TRACK_TAG_PACKER(track_type, track_chunk) ### JLS_DATATYPE_BASETYPE_INT ### JLS_DATATYPE_BASETYPE_UNSIGNED ### JLS_DATATYPE_BASETYPE_UINT ### JLS_DATATYPE_BASETYPE_FLOAT ### JLS_DATATYPE_DEF(basetype, size, q) Construct a JLS datatype. * **Parameters:** * **basetype** – The datatype base type, one of [INT, UINT, FLOAT, BOOL] * **size** – The size in bits. Only the following options are supported: * INT: 4, 8, 16, 24, 32, 64 * UINT: 1, 4, 8, 16, 24, 32, 64 * FLOAT: 32, 64 * BOOL = UINT 1 * **q** – The signed fixed-point location, only valid for INT and UINT. Set to 0 for normal, whole numbers. Set to 0 for FLOAT and BOOL. The fixed point is between bits q and q-1. The value is scaled by 2 \*\* -q. ### JLS_DATATYPE_I4 ### JLS_DATATYPE_I8 ### JLS_DATATYPE_I16 ### JLS_DATATYPE_I24 ### JLS_DATATYPE_I32 ### JLS_DATATYPE_I64 ### JLS_DATATYPE_U1 ### JLS_DATATYPE_U4 ### JLS_DATATYPE_U8 ### JLS_DATATYPE_U16 ### JLS_DATATYPE_U24 ### JLS_DATATYPE_U32 ### JLS_DATATYPE_U64 ### JLS_DATATYPE_BOOL ### JLS_DATATYPE_F32 ### JLS_DATATYPE_F64 ### Enums ### enum jls_signal_type_e The signal type definition. *Values:* ### enumerator JLS_SIGNAL_TYPE_FSR Fixed sampling rate. ### enumerator JLS_SIGNAL_TYPE_VSR Variable sampling rate. ### enum jls_track_type_e The available track types that store data over time. *Values:* ### enumerator JLS_TRACK_TYPE_FSR Block tracks contain fixed sample-rate (FSR) data. The JLS_TAG_SIGNAL_DEF defines the sampling rate. ### enumerator JLS_TRACK_TYPE_VSR Fixed-type, variable-sample-rate (VSR) time series data. Each data entry consists of time in UTC and the data. ### enumerator JLS_TRACK_TYPE_ANNOTATION Annotations contain infrequent, variable-typed data. Each annotation data entry consists of time and associated annotation data. For FSR, time must be samples_id. For VSR, time must be UTC. #### SEE ALSO [jls_annotation_type_e](#group__jls__format_1ga4fd2e5bb01666983ac6183a7c74cf7b0) for the annotation types. ### enumerator JLS_TRACK_TYPE_UTC The UTC track associates sample_id with UTC. Each utc data entry consists of sample_id, UTC timestamp pairs. This track is only used for FSR signals. ### enumerator JLS_TRACK_TYPE_COUNT The total number of track types. ### enum jls_storage_type_e The data storage type. The data storage type applies to directly user-accessible data including annotations and user_data. *Values:* ### enumerator JLS_STORAGE_TYPE_INVALID Invalid (unknown) storage type. ### enumerator JLS_STORAGE_TYPE_BINARY Raw binary data. ### enumerator JLS_STORAGE_TYPE_STRING Null-terminated C-style string with UTF-8 encoding. ### enumerator JLS_STORAGE_TYPE_JSON JSON serialized data structure with NULL terminator and UTF-8 encoding. ### enum jls_annotation_type_e The available annotation types. *Values:* ### enumerator JLS_ANNOTATION_TYPE_USER Arbitrary user data. Application-dependent data with no standardized form or purpose. ### enumerator JLS_ANNOTATION_TYPE_TEXT UTF-8 formatted text. Viewers should display this text at the appropriate location. The [jls_storage_type_e](#group__jls__format_1ga8b5a75849f337b751c2b844039c45e4a) must be STRING. ### enumerator JLS_ANNOTATION_TYPE_VERTICAL_MARKER A vertical marker at a given time. Marker names can be arbitrary, but the convention is: * Number strings, like “1”, represent a single marker. * Number + a/b string, like “1a” and “1b”, represent a marker pair (dual markers). The name should be UTF-8 encoded. The marker name can be followed by file separator ASCII 28 (0x1C) and then followed by application-specific JSON-formatted metadata. The [jls_storage_type_e](#group__jls__format_1ga8b5a75849f337b751c2b844039c45e4a) must be STRING. ### enumerator JLS_ANNOTATION_TYPE_HORIZONTAL_MARKER A horizontal marker at a given y-axis value. Marker names can be arbitrary, but the convention is: * Number strings, like “1”, represent a single marker. * Number string + a/b string, like “1a” and “1b”, represent a marker pair (dual markers). The [jls_storage_type_e](#group__jls__format_1ga8b5a75849f337b751c2b844039c45e4a) must be STRING. ### enum jls_track_chunk_e The chunks used to store the track information. *Values:* ### enumerator JLS_TRACK_CHUNK_DEF Track definition chunk. This chunk contains a zero-length (empty) payload. ### enumerator JLS_TRACK_CHUNK_HEAD Track offsets for the first chunk at each level. This chunk provides fast seek access to the first chunk at each summary level for this track. The payload is [jls_track_head_s](#structjls__track__head__s) which contains the offset to the first INDEX chunk. #### SEE ALSO [jls_track_head_s](#structjls__track__head__s) for all track types. ### enumerator JLS_TRACK_CHUNK_DATA The data chunk. The payload varies by track type and data format. All DATA payloads start with [jls_payload_header_s](#structjls__payload__header__s). #### SEE ALSO jls_fsr_f32_data_s for FSR float32. #### SEE ALSO [jls_annotation_s](#structjls__annotation__s) for ANNOTATION. #### SEE ALSO [jls_utc_data_s](#structjls__utc__data__s) for UTC. ### enumerator JLS_TRACK_CHUNK_INDEX Provides the timestamp and offset for each contributing data chunk. This chunk MUST be immediately follow by a SUMMARY chunk. All INDEX payloads start with [jls_payload_header_s](#structjls__payload__header__s). #### SEE ALSO [jls_fsr_index_s](#structjls__fsr__index__s) for FSR track types. #### SEE ALSO [jls_index_s](#structjls__index__s) for all other track types. ### enumerator JLS_TRACK_CHUNK_SUMMARY The summary chunk. The payload format is defined by the track type. All CHUNK payloads start with [jls_payload_header_s](#structjls__payload__header__s). #### SEE ALSO [jls_fsr_f32_summary_s](#structjls__fsr__f32__summary__s) for FSR float32. #### SEE ALSO [jls_annotation_summary_s](#structjls__annotation__summary__s) for ANNOTATION. #### SEE ALSO [jls_utc_summary_s](#structjls__utc__summary__s) for UTC. ### enum jls_tag_e The tag definitions. *Values:* ### enumerator JLS_TAG_INVALID ### enumerator JLS_TAG_SOURCE_DEF ### enumerator JLS_TAG_SIGNAL_DEF ### enumerator JLS_TAG_TRACK_FSR_DEF ### enumerator JLS_TAG_TRACK_FSR_HEAD ### enumerator JLS_TAG_TRACK_FSR_DATA ### enumerator JLS_TAG_TRACK_FSR_INDEX ### enumerator JLS_TAG_TRACK_FSR_SUMMARY ### enumerator JLS_TAG_TRACK_VSR_DEF ### enumerator JLS_TAG_TRACK_VSR_HEAD ### enumerator JLS_TAG_TRACK_VSR_DATA ### enumerator JLS_TAG_TRACK_VSR_INDEX ### enumerator JLS_TAG_TRACK_VSR_SUMMARY ### enumerator JLS_TAG_TRACK_ANNOTATION_DEF ### enumerator JLS_TAG_TRACK_ANNOTATION_HEAD ### enumerator JLS_TAG_TRACK_ANNOTATION_DATA ### enumerator JLS_TAG_TRACK_ANNOTATION_INDEX ### enumerator JLS_TAG_TRACK_ANNOTATION_SUMMARY ### enumerator JLS_TAG_TRACK_UTC_DEF ### enumerator JLS_TAG_TRACK_UTC_HEAD ### enumerator JLS_TAG_TRACK_UTC_DATA ### enumerator JLS_TAG_TRACK_UTC_INDEX ### enumerator JLS_TAG_TRACK_UTC_SUMMARY ### enumerator JLS_TAG_USER_DATA ### enumerator JLS_TAG_END ### enum jls_summary_fsr_e The summary storage order for each entry. Entries hold raw values: readers scale fixed-point (q != 0) signals by 2^-q, see JLS_DATATYPE_DEF. *Values:* ### enumerator JLS_SUMMARY_FSR_MEAN ### enumerator JLS_SUMMARY_FSR_STD ### enumerator JLS_SUMMARY_FSR_MIN ### enumerator JLS_SUMMARY_FSR_MAX ### enumerator JLS_SUMMARY_FSR_COUNT ### Functions ### static inline uint8_t jls_datatype_parse_basetype(uint32_t dt) ### static inline uint8_t jls_datatype_parse_size(uint32_t dt) ### static inline uint8_t jls_datatype_parse_q(uint32_t dt) ### struct jls_source_def_s *#include * The source definition. ### Public Members ### uint16_t source_id The source identifier. 0 reserved for global annotations, must be unique per instance ### const char \*name The source name string. ### const char \*vendor The vendor name string. ### const char \*model The model string. ### const char \*version The version string. ### const char \*serial_number The serial number string. ### struct jls_signal_def_s *#include * The signal definition. On write, zero values select per-data-type defaults. The writer then aligns the FSR parameters: sample_decimate_factor >= 10 and a multiple of 256 bits of samples, samples_per_data a multiple of sample_decimate_factor, entries_per_summary a multiple of both summary_decimate_factor (>= 10) and the entries per data chunk, and annotation_decimate_factor and utc_decimate_factor >= 10. Caps: the writer rejects definitions that exceed them after alignment, and the reader skips FSR signals whose definitions violate them: * samples_per_data <= 2^22 (4 Mi samples), * samples_per_data \* sample size <= 16 MiB of data per chunk, * entries_per_summary <= 2^19 (16 MiB of 64-bit summaries), * annotation_decimate_factor and utc_decimate_factor <= 2^16 (write). On read, an FSR definition must also have sample_decimate_factor dividing samples_per_data, a whole number of bytes per summary entry, entries_per_summary a multiple of the entries per data chunk and of summary_decimate_factor, and summary_decimate_factor >= 2. ### Public Members ### uint16_t signal_id The signal identifier. 0 to JLS_SIGNAL_COUNT - 1, must be unique per instance ### uint16_t source_id The source identifier, must match a source_def. ### uint8_t signal_type The [jls_signal_type_e](#group__jls__format_1gaa4559f0739dd1cb5509632ee88b9bfbd) signal type. ### uint32_t data_type The JLS_DATATYPE_\* data type for this signal. ### uint32_t sample_rate TThe sample rate per second (Hz). 0 for VSR. ### uint32_t samples_per_data The number of samples per data chunk. (write suggestion). ### uint32_t sample_decimate_factor The number of samples per summary level 1 entry. ### uint32_t entries_per_summary The number of entries per summary chunk. (write suggestion). ### uint32_t summary_decimate_factor The number of summaries per summary, level >= 2. ### uint32_t annotation_decimate_factor The annotation decimate factor for summaries. ### uint32_t utc_decimate_factor The UTC decimate factor for summaries. ### int64_t sample_id_offset The sample id offset for the first sample. (FSR only). ### const char \*name The signal name. While C supports all possible names, language bindings may simplify signal lookup. The Python pyjls package allows lookup by “source_name.signal_name” and “signal_id”. For full compatibility: 1. Avoid names convertable to integers 2. Avoid names with ‘.’ characters 3. Use ASCII or valid UTF-8 encoding ### const char \*units The units string, normally as SI with no scale prefix. ### struct jls_track_head_s *#include * The track head payload for JLS_TRACK_CHUNK_HEAD. ### union jls_version_u *#include * Union structure for parsing 32-bit versions. ### Public Members ### uint32_t u32 ### uint16_t patch ### uint8_t minor ### uint8_t major ### struct [jls_version_u](#_CPPv413jls_version_u)::[anonymous] s ### struct jls_file_header_s *#include * The JLS file header structure. ### Public Members ### uint8_t identification[16] The semi-unique file identification header. #### SEE ALSO [JLS_HEADER_IDENTIFICATION](#group__jls__format_1gabff27d15b22a3c03133fda96ea91cb72) ### uint64_t length The file length in bytes. This is the last field updated on file close. If the file was not closed gracefully, the value will be 0. ### union [jls_version_u](#_CPPv413jls_version_u) version The JLS file format version number. #### SEE ALSO JLS_FORMAT_VERSION_U32 ### uint32_t crc32 The CRC32 from the start of the file through version. ### struct jls_chunk_header_s *#include * The JLS chunk header structure. Every chunk starts with this header. This header identifies the chunk’s payload, and enables the chunk to participate in one doubly-linked list. If the length is zero, then the next chuck header immediately follows this chunk header. If the length is not zero, then the chunk consists of: * A chunk header * payload of length bytes * Zero padding of 0-7 bytes, so that the entire chunk will end on a multiple of 8 bytes. This field ends on: 8 \* k - 4 * crc32 over the payload. ### Public Members ### uint64_t item_next The next item. The chunk header enables each chunk to participate in a single, doubly-linked list. This field indicates the location for the next item in the list. The value is relative to start of the file. 0 indicates end of list. This field allows simple, linear traversal of data, but the next chunk is not known when the chunk is first created. Therefore, this field requires that chunk headers are updated when the software writes the next item to the file. ### uint64_t item_prev The previous item. The chunk header enables each chunk to participate in a single, doubly-linked list. This field indicates the location for the previous item in the list. The value is relative to start of the file. 0 indicates start of list. ### uint8_t tag The tag. The [jls_tag_e](#group__jls__format_1ga17fd831049e87407c8206463ea75ff10) value that identifies the contents of this chunk. ### uint8_t rsv0_u8 Reserved for future use. (compression?). ### uint16_t chunk_meta The metadata associated with this chunk. Each tag is free to define the purpose of this field. However, all data tags use this definition: * chunk_meta[7:0] is the signal/time series identifier from 0 to 255. * chunk_meta[11:8] is reserved. * chunk_meta[15:12] contains the depth for this chunk from 0 to 15. * 0 = block (sample level) * 1 = First-level summary of block samples * 2 = Second-level summary of first-level summaries. User-data reserves chunk_meta[15:12] to store the storage_type and another internal indications. chunk_meta[11:0] may be assigned by the specific application. ### uint32_t payload_length The length of the payload in bytes. Can be 0. In addition to defining the payload size, this value is also used for forward chunk traversal. ### uint32_t payload_prev_length The length of the previous payload in bytes. Used for reverse chunk traversal. ### uint32_t crc32 The CRC32 over the header, excluding this field. ### struct jls_payload_header_s *#include * The payload header for DATA, INDEX, and SUMMARY chunks. ### Public Members ### int64_t timestamp The sample_id for the first entry. ### uint32_t entry_count The total number of entries. ### uint16_t entry_size_bits The size of each entry, in bits. ### uint16_t rsv16 Reserved. ### struct jls_fsr_data_s *#include * The FSR data chunk format. ### Public Members ### struct [jls_payload_header_s](#_CPPv420jls_payload_header_s) header The payload header. ### float data[] The summary data. Although data’s type is float, the actual data type depends upon the signal definition. ### struct jls_fsr_index_s *#include * The payload for JLS_TAG_TRACK_FSR_INDEX chunks. Since FSR has a fixed sample rate, the header contains enough information to fully identify the timestamp for each offset. Therefore, no additional time information is required per entry. #### SEE ALSO [jls_index_s](#structjls__index__s) for all other INDEX chunk types. ### Public Members ### uint64_t offsets[] The chunk file offsets, spaced by fixed time intervals. ### struct jls_fsr_f32_summary_s *#include * The FSR summary chunk format with f32 values. This summary format is used by all types except u32, u64, i32, i64, f64. ### Public Members ### struct [jls_payload_header_s](#_CPPv420jls_payload_header_s) header The payload. ### float data[][[JLS_SUMMARY_FSR_COUNT](#_CPPv4N17jls_summary_fsr_e21JLS_SUMMARY_FSR_COUNTE)] The summary data, each entry is 4 x f32: mean, std, min, max. ### struct jls_fsr_f64_summary_s *#include * The FSR summary chunk format with f64 values. This summary format is used by u32, u64, i32, i64, f64. ### Public Members ### struct [jls_payload_header_s](#_CPPv420jls_payload_header_s) header The payload. ### double data[][[JLS_SUMMARY_FSR_COUNT](#_CPPv4N17jls_summary_fsr_e21JLS_SUMMARY_FSR_COUNTE)] The summary data, each entry is 4 x f64: mean, std, min, max. ### struct jls_index_entry_s *#include * The entry format for JLS_TRACK_CHUNK_INDEX payloads. #### SEE ALSO [jls_index_s](#structjls__index__s) ### Public Members ### int64_t timestamp The timestamp for this entry. sample_id for FSR, UTC for VSR. ### uint64_t offset The chunk file offset. ### struct jls_index_s *#include * The payload for JLS_TRACK_CHUNK_INDEX chunks. The INDEX payload maps timestamps to offsets to allow fast seek. However, the JLS_TAG_TRACK_FSR_INDEX uses the [jls_fsr_index_s](#structjls__fsr__index__s) since the timestamp provides unnecessary, duplicative information for FSR tracks. #### SEE ALSO [jls_fsr_index_s](#structjls__fsr__index__s) for JLS_TAG_TRACK_FSR_INDEX. ### struct jls_annotation_s *#include * Hold a single annotation record. This structure is used by both the API and JLS_TAG_TRACK_ANNOTATION_DATA. ### Public Members ### int64_t timestamp The timestamp for this annotation. sample_id for FSR, UTC for VSR. ### uint64_t rsv64_1 Reserved, write to 0. ### uint8_t annotation_type The [jls_annotation_type_e](#group__jls__format_1ga4fd2e5bb01666983ac6183a7c74cf7b0). ### uint8_t storage_type The [jls_storage_type_e](#group__jls__format_1ga8b5a75849f337b751c2b844039c45e4a). ### uint8_t group_id The optional group identifier. If unused, write to 0. ### uint8_t rsv8_1 Reserved, write to 0. ### float y The y-axis value or NAN to automatically position. ### uint32_t data_size The size of data in bytes. ### uint8_t data[] The annotation data. ### struct jls_annotation_summary_entry_s *#include * The entry format for JLS_TAG_TRACK_ANNOTATION_SUMMARY. #### SEE ALSO [jls_annotation_summary_s](#structjls__annotation__summary__s) ### Public Members ### int64_t timestamp The timestamp (duplicates INDEX). ### uint8_t annotation_type The [jls_annotation_s.annotation_type](#structjls__annotation__s_1a4340d4cfeeac6e50f3869a8a58d2d5c5). ### uint8_t group_id The [jls_annotation_s.group_id](#structjls__annotation__s_1aa0301ead8cbed2d14d44d0a66bdf0f84). ### uint8_t rsv8_1 Reserved, write to 0. ### uint8_t rsv8_2 Reserved, write to 0. ### float y The [jls_annotation_s.y](#structjls__annotation__s_1aa4f0d3eebc3c443f9be81bf48561a217). ### struct jls_annotation_summary_s *#include * The payload format for JLS_TAG_TRACK_ANNOTATION_SUMMARY chunks. ### struct jls_utc_data_s *#include * The entry format for JLS_TAG_TRACK_UTC_DATA. This same format is reused for summary entries. UTC.DATA only exists to provide recovery in the event that the file is not properly closed. #### SEE ALSO [jls_utc_summary_s](#structjls__utc__summary__s) ### Public Members ### int64_t timestamp The timestamp in UTC. ### struct jls_utc_summary_entry_s *#include * The entry format for JLS_TAG_TRACK_UTC_SUMMARY. This same format is reused for summary entries. UTC.DATA only exists to provide recovery in the event that the file is not properly closed. #### SEE ALSO [jls_utc_summary_s](#structjls__utc__summary__s) ### Public Members ### int64_t sample_id The timestamp in sample ids (duplicates INDEX). ### int64_t timestamp The timestamp in UTC. ### struct jls_utc_summary_s *#include * The payload format for JLS_TAG_TRACK_UTC_SUMMARY chunks.