background
for now, we had design a file server from db to s3 and mf reader to feed data to the ads test module, which run in SIMD mode, e.g. k8s. for a closed-loop SIL, another component is how to record test message/outputs to database.
I am thinking firstly event driven webframework python, or plusar, which is a popular python event-driven project. as lgsvl pythonAPI is really a good choice to transfer message among server and clients. so the basic idea is to integrate websocket/tcp channel inside aeb model
websocket vs tcp
most ADAS models developed in OEM are matlab based, and the most common communication protocol in Matlab is UDP, a little review about tcp vs udp, I personally don’t like udp, no reason. so narrow to TCP, the further question is: should it be a raw TCP or Websocket), websocket enables a stream of messages instead of a stream of bytes. WebSockets is built on normal TCP sockets and uses frame headers that contains the size of each frame and indicate which frames are part of a message. The WebSocket API re-assembles the TCP chunks of data into frames which are assembled into messages before invoking the message event handler once per message. WebSocket is basically an application protocol (with reference to the ISO/OSI network stack), message-oriented, which makes use of TCP as transport layer. btw, TCP is bidirectional because it can send data in both directions, and it is full-duplex because it can do that simultaneously, without requiring line turnarounds, at the API level
for ADAS test outputs, it’s plain txt, I’d prefer websocket. so make the decision.
matlab websocket client
there is an github project: matlab websocket, which is enough to implement a simple websocket in Matlab. discussion at stackoverflow
basically, here we bind a weboscket client to the adas model, in matlab. which is a good choice, so when runnign adas model in massively, the whole system looks like multi ws clients concurrently communicate with one ws server, which is well separated from adas model, and can freely implmented in either nodejs, python or c# e.t.c., totally friendly to other third-party data analysis tools in down-stream.
multi-write db concurrently
once we get the message from ws clients, we need transfer all of them to a database, e.g. pySQL. but the first thing need to understand is whether SQL can concurrent inserts. if there are multiple INSERT statements, they are queued and performed in sequence, concurrently with the SELECT statements. SQL doesn’t support parallel data inserts into the same table, with InnoDB or MyISAM storage engine.
this also answered another question, why not directly adas model write to db, since the parallel runinng cases increase, writing DB will be the bottleneck. so has the websocket middle level, is a good choice.
nodejs solution
nodejs is really powerful to implement the solution above. basically, a ws server and sql writer: mysqljs
python solution
similar as nodejs, python has matured websocket module and ORM moduel to write db.