[Engine] Adopt Ray as orchestration backend #14
Labels
Core
Core functionality that impacts the engine design
Engine
The work is on the engine side
type: feature
A feature to be implemented
Milestone
I've been experimenting with the pipeline using native multiprocessing and Redis Streams/RQ recently, and it quickly becomes messy when we spawn many processes.
So I'm evaluating Ray as the backend engine to orchestrate the streaming processing jobs while supporting batch learning that anomaly detection might utilize. By far, it looks promising.
The main benefit of Ray to us includes:
@Liangshumin @Fengrui-Liu FYI, there'll be some changes to the existing designs that I communicated over chat, please pay attention to the algorithm training part as Ray offers many out-of-the-box ML features.
The text was updated successfully, but these errors were encountered: