Skip to content

Understanding LazyFutureStreams behaviour

johnmcclean-aol edited this page Mar 20, 2015 · 1 revision

The most complex implementation

The most complex of the three Stream types in simple-react is the LazyFutureStream. This Stream allows a potentially infinite amount of data to pass through, and so must behave slightly differently than an EagerFutureStream.

A simple example

A simple example of this is the limit function - on an EagerFutureStream we can allow all processing to start and limit access to the next phase to the first 2 results. With a LazyFutureStream that could be infinite in size it is not feasible to allow all processing to start - so instead we must limit the start of processing.

Internal queues

All three types of Streams use async.Queues internally as part of the implementation of some functionality (for SimpleReactStream this only applies to flatMap). A Queue is populated asynchrnously and the next phase of the stream is read from it as data comes in.

Bounded and unbounded queues

EagerFutureStreams and SimpleReactStreams use unbounded Queues by default. For eager processing data set must be limited by definition. LazyFutureStream which can handle infinite processing uses bounded Queues by default. When the bounded Queue is full, it will block the populating thread. For this reason LazyFutureStreams use a separate ExecutorService is used for each Queue population phase.

Populating queues

Because populating threads can block, it is also important that any events that close the LazyFutureStream are propagated to the full Queues and their blocked producers - so that they can stop and free themselves up for garbage collection. Prior to v0.70 of simple-react for a non-infinite (i.e. limited) LazyFutureStream users had to manage this process themselves, by setting limits early in the stream definition. As of v0.70 simple-react will close and notify any Queue's and their producers for you. simple-react will also stop any infinite generators populating the stream.

lazyfuturestream flatmap - toqueue

Clone this wiki locally