> Sematic makes I/O between steps in your pipelines as simple as passing an output of one python function as the input of another. Airflow provides APIs which can pass data between tasks, but involves some boilerplate around explicitly pushing/pulling data around, and coupling producers and consumers via named data keys.
In robotics you sometimes need high performance data transformation e.g. convert pile of raw robot log data protos --> pile of simulation inputs --> pile of extracted data --> munged into net input format
Does semantic support this if the communication between tasks uses python functions? Like if my simulator is C++, will I have to use SWIG?
In some of the competing systems, the input/output between nodes are just produced files as side effects, which is nice because it doesn't care what language / infra you use as long as you produce the required input/output.
What we have seen done in the past is use things like pybind11 to expose C++ APIs in Python, which I guess is a similar concept to SWIG. If you are using build tools such as Bazel, you can even get the C++ compiled at run-time when submitting your pipeline.
Regarding i/o artifacts, Sematic lets user choose how they are serialized. We offer reasonable baseline defaults, but certain artifacts require serialization formats that are cross-language (e.g. ROS messages since you mention robotics).
As a last resort, users are free to serialize and persist artifacts by hand as part of their pipeline functions (e.g. storing in a cloud bucket) and only returning a reference to said artifact (e.g. a Python dataclass with artifact location and metadata).
> Sematic makes I/O between steps in your pipelines as simple as passing an output of one python function as the input of another. Airflow provides APIs which can pass data between tasks, but involves some boilerplate around explicitly pushing/pulling data around, and coupling producers and consumers via named data keys.
In robotics you sometimes need high performance data transformation e.g. convert pile of raw robot log data protos --> pile of simulation inputs --> pile of extracted data --> munged into net input format
Does semantic support this if the communication between tasks uses python functions? Like if my simulator is C++, will I have to use SWIG?
In some of the competing systems, the input/output between nodes are just produced files as side effects, which is nice because it doesn't care what language / infra you use as long as you produce the required input/output.