New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

StreamRefs #3321

Merged

marcpiechura merged 10 commits into akkadotnet:v1.4 from Horusiath:stream-refs

Feb 24, 2018

Contributor

Horusiath commented Feb 10, 2018 •

edited

Loading

This is work in progress over porting StreamRefs feature to .NET.

TODOs:

Ported settings, and all stream-ref related types
StreamRefSpec (except serialization)
Serialization (in progress) - this is the most problematic part, thanks to awesome support of generic data types by Protobuf. Also I've noticed, that JVM implementation gave up on serializing exceptions. I think we could go a little more ambitious here.
Documentation

This PR may require akka.streams 2.5 to be merged first (cc: @marcpiechura).

Aaronontheweb added the needs review label

Contributor

marcpiechura commented Feb 10, 2018

@Horusiath anything in particular ?
Progress on 2.5 is tracked here #2601 (comment) but I‘ve moved most of the tasks to #2552 since they depend on the new materializer

Contributor Author

Horusiath commented Feb 13, 2018

@marcpiechura I think that there were quite big changes in a way how stage actors are created. From what I've seen, it's no longer getStageActorRef()

Contributor

marcpiechura commented Feb 13, 2018

@Horusiath I think it’s this PR akka/akka#19487 but it’s quite old, not sure why we haven’t implemented it already

Horusiath mentioned this pull request

FunctionRef #3324

Merged

Horusiath added 4 commits

February 21, 2018 20:35


          initial commit for StreamRefs

cfac9f0


          more work over Stream-Ref serializer

360c0cf


          fixes in serializer and config

08c2d64


          fixes in stream-refs and tests

Horusiath force-pushed the stream-refs branch from 8873fd0 to 5493130 Compare

February 21, 2018 21:25

Horusiath changed the base branch from dev to v1.4

February 21, 2018 21:25


          StreamRefs docs

241d710

Contributor Author

Horusiath commented Feb 23, 2018

If this CI will pass, this is ready for review and merge. I haven't implemented exception stack trace propagation over the wire, but it can be added even after the release.

Contributor

marcpiechura commented Feb 23, 2018

@Horusiath looks like you forgot to set the config parameter in the reference configuration for the tests ?!

System.ArgumentNullException : `akka.stream.materializer.stream-ref` was not present
Parameter name: config
   at Akka.Streams.Implementation.StreamRefSettings.Create(Config config)
   at Akka.Streams.ActorMaterializerSettings.Create(Config config)
   at Akka.Streams.Tests.Dsl.ActorRefSourceSpec..ctor(


          added defaults for StreamRefsSettings

efcea83

Horusiath changed the title ~~[WIP] StreamRefs~~ StreamRefs

Contributor Author

Horusiath commented Feb 24, 2018

@marcpiechura for some reason, when no config is provided, a reference.conf from Akka.Streams is not applied.


          StreamRefs approvals API

8d62cb3

Contributor

marcpiechura commented Feb 24, 2018

@Horusiath One spec is still failing

StreamRefsSpec.SinkRef_must_receive_hundreds_of_elements_via_remoting
Failed: Expected a message of type System.String, but received {Failure: System.ArgumentException: Cannot push port twice SourceRef.out

Contributor Author

Horusiath commented Feb 24, 2018

It's racy behavior, working on it.

marcpiechura suggested changes

View reviewed changes

docs/articles/streams/streamrefs.md Outdated


		The process of preparing and running a `ISourceRef<T>` powered distributed stream is shown by the animation below:

		![]()

Contributor

marcpiechura Feb 24, 2018

Missing link to the animations

docs/articles/streams/streamrefs.md Outdated


		The process of preparing and running a `ISinkRef<>` powered distributed stream is shown by the animation below:

		![]()

Contributor

marcpiechura Feb 24, 2018

As above

src/core/Akka.Streams/Dsl/StreamRefs.cs Outdated

+                  {
+                      public CumulativeDemand(long seqNr)
+                      {
+                          if (seqNr < 0) throw ReactiveStreamsCompliance.NumberOfElementsInRequestMustBePositiveException;

Contributor

marcpiechura Feb 24, 2018

Should be <=

src/core/Akka.Streams/Dsl/StreamRefs.cs

+                      private readonly EnumerableActorName sourceRefStageNames = new EnumerableActorNameImpl("SourceRef", new AtomicCounterLong(0L));
+                      private readonly EnumerableActorName sinkRefStageNames = new EnumerableActorNameImpl("SinkRef", new AtomicCounterLong(0L));
+                      public StreamRefsMaster(ExtendedActorSystem system)

Contributor

marcpiechura Feb 24, 2018

Can the parameter be removed since it’s not stored ?

Contributor Author

Horusiath Feb 24, 2018

I just want to keep the convention for extensions here.

src/core/Akka.Streams/Dsl/StreamRefs.cs

+                  internal sealed class SourceRefImpl<T> : SourceRefImpl, ISourceRef<T>
+                  {
+                      public SourceRefImpl(IActorRef initialPartnerRef) : base(initialPartnerRef) { }
+                      public override Type EventType => typeof(T);

Contributor

marcpiechura Feb 24, 2018

Wouldn’t be a single assignment better ?

Contributor Author

Horusiath Feb 24, 2018

It's used only for deserialization (usually once per instance per node). I don't want to allocate underlying field for it, as Hyperion/Json.NET are able to provide generic type param directly from SourceRefImpl<T>'s T parameter, so the payload will be smaller.

src/core/Akka.Streams/Dsl/StreamRefs.cs

+                  internal sealed class SinkRefImpl<T> : SinkRefImpl, ISinkRef<T>
+                  {
+                      public SinkRefImpl(IActorRef initialPartnerRef) : base(initialPartnerRef) { }
+                      public override Type EventType => typeof(T);

Contributor

marcpiechura Feb 24, 2018

As above

src/core/Akka.Streams/Dsl/StreamRefs.cs Outdated

+                              this.SetHandler(_stage.Inlet,
+                                  onPush: OnPush,
+                                  onUpstreamFinish: OnUpstreamFinish,
+                                  onUpstreamFailure: OnUpstreamFailure);

Contributor

marcpiechura Feb 24, 2018

Implement IInHandler and use SetHandler(_stage.Inlet, this)

src/core/Akka.Streams/Dsl/StreamRefs.cs Outdated

+                              _promise = promise;
+                              _inheritedAttributes = inheritedAttributes;
+                              SetHandler(_stage.Outlet, onPull: OnPull);

Contributor

marcpiechura Feb 24, 2018

Implement IOutHandler


          fixed missing impl + added animation gifs to docs

marcpiechura approved these changes

View reviewed changes

marcpiechura reviewed

View reviewed changes

src/core/Akka.Streams/Dsl/StreamRefs.cs Outdated

+                          public void OnDownstreamFinish()
+                          {
+                              /* IOutHandler impl */

Contributor

marcpiechura Feb 24, 2018

Needs to call CompleteStage

Horusiath added 2 commits

February 24, 2018 19:27


          applied docs, fixed namespaces

8d61fd8


          fixed stream ref serializer namespaces

2d54f62

marcpiechura merged commit 184e7c7 into akkadotnet:v1.4

marcpiechura mentioned this pull request

Akka.Net Streams and remoting (Sink.ActorRefWithAck) #3371

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels