mdf4 reader

read mdf4 in general

the most important “branch” in the tree is the list of data groups (DG block).

record used to store the plain measurement data, the records can either be contianed in one single “data” (DT) block, or distributed over several DT blocks using a “data list” block .

Each DG block has points to the data block, as well as necessary information to understand and decode the data.

the sub-tree of DG is channel group(CG) blocks and channel(CN) blocks.

record layout

each record contains all signal values which have the same time stamp, in the same channel. for different channel groups(CG), each record must start with a “record id”

the record layout of ID 1, will be described by one channel group, the record layout of ID 2 by another channel group. Both channel groups must be children of the DT block’s parent data group.

channels

Each channel describes a recorded signal and how its values are encoded in the records for the channel’s parent CG block.

vector APIs

which is licensed by vector

• GetFileManager()
• OpenFile()
• GetChannelByName()
• GetChannelGroup()
• CreaterDataPointer()
• GetRecordCount()
• SetReadMode()
• SeekPos()
• GetPos()
• GetTimeSec()
• ReadPhysValueDouble(pChannle, &value, &isvalid)
• ReadRawValueDouble(pChannel, &rawValue)
• ReadPhysValueDouble(pChannel, &phyValue)
• MultiReadPhysValueDouble(pChannel, valueBuffer, size, &read)

turbolab

ReadFileMF4 class:

• Open()
• Seek(pos)
• Read(Size, Buffer)
• Close()

CMdf4Calc class :

• Cmdf4Calc(pChan, pblock)
• Cmdf4Calc(pChan, mf4file)

cmf4DataGroup class:

• GetRecord()
• GetRawValueFromRecord()
• GetValueFromRecord()

mdf4FileImport class :

• ImportFile()
• ReleaseFile()

I am not reading this C++ code.

asammdf

MDF class

• init(name=None, ): name can be either string or BytesIO
• filter(channels, )
• get_group(index, channels=None, )
• iter_channels(,)
• iter_get(channel_name=None,group_id=None, group_index=None, )
• iter_groups()
• to_dataframe(), generate pandas DataFrame
• whereis(channel_name)

MDF4 class

includes data_group, channel_group, channels, data_block, data_location, record_size e.t.c

• get(channel_name=None, group=None, )
• get_can_signal()
• get_channel_name(group, index)
• get_channel_unit(channel_name=Nont, group=None, )
• get_master(index, data=None, ), return the master channel samples for given group
• info()

the following are sub data blocks of MDF4:

• Channel class
• ChannelGroup class
• DataGroup class
• DataList class
• DataBlock class
• HeaderBlock class

mdf = MDF4("basic.mf4")
db_info = mdf.info()

all group, channel data is packages as python dict. the most exciting part, as asam-mdf can support BytesIO, which gives the way to read mdf files stores in Amazon S3:

    mf4_ = obj_api.get_object("bucket_name", "sample.mf4")
    fid_ = io.BufferedReader(mf4_['Body']._raw_stream) ;
    read_in_memory = fid_.read()
    bio_ = io.BytesIO(read_in_memory);
    if bio_.seekable(): 
        mdf_ = MDF(bio_)
        mdf_.info()
``` 

### installation of asam-mdf

* [install pandas](https://www.pypandas.cn/en/docs/installation.html#installing-using-your-linux-distribution’s-package-manager)

`python3 -m pip install pandas` (not working)

`sudo apt-get install python3-pandas` 

* [install canmatrix](https://canmatrix.readthedocs.io/en/latest/installation.html)

`sudo python3 setup.py install` 

* install pathlib2 

`sudo -H pip3 install pathlib2 `

if `WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.`
using: `sudo -H python3 -m pip install（包名）`


## [mdfreader](https://github.com/ratal/mdfreader)

which is open source LSP licensed. and the APIs are clear about getChannels, and read data from each channel, which was my first choice. but later found out it's difficult to support streamIO input, which was the best format I can reached from Amazon S3 pipeline.


##### MdfInfo

* read_info()
* list_channels()

##### Mdf

* read()
* filter_channel_names()
* get_channel_data()
* get_channel_unit()
* keep_channels()

```python
import mdfreader
filename="basic.mf4"
file_info=mdfreader.MdfInfo()
channels = file_info.list_channels(filename)
print(channels)
file_pointer = mdfreader.Mdf(filename)
keys_ = file_pointer.keys()
for k_ in keys_:
    chan1 = file_pointer.get_channel_data(k_)
    print(chan1)

python necessary

buffered io

binary I/O, usually used in following:

f = open("mdf_file", "rb")
type(f)
<type '_io.BufferedReader'>

open(file, mode=’r’, buffering=-1, encoding=…)

ifmode='b', the returned content is bytes object, can’t encoding. if buffereing is not setted, for binary file with buffering with a fixed length. when with mode='w/r/t, it returns io.TextIOwrapper, if mode=rb, it returns io.BufferedReader, else if mode=wb, it returns io.BufferedWriter, else mode=wrb, it returns io.BufferedRandom. check here

dict built-in funs

cmp(dict1, dict2) ;
len(dict)
str(dict)
dict.clear()
dict.get(key)
dict.has_key(key)
dict.items()
dict.keys()
dict.values()
pop(key)

a bit about readers

a few years ago I was in CAE field, dealing with different kind of CAE format, e.g .nas. during that time, there is a similar need to have a file transfer from vendor specified format to python format e.t.c.

so now it comes again, which is funny to know adapters in CAE and autonomous driving design.

refer

asammdf API doc

asam mdf4 doc