review changes in akka-projections 1.3.0 (available now as Apache License, 2.0)

[pjfanning]

3 years since release

https://github.com/akka/akka-projection/commits/v1.3.0/
https://github.com/akka/akka-projection/releases/tag/v1.3.1

• Query plugin paramater in source providers akka/akka-projection#684
• akka/akka-projection@708066a
• fix: handle DeletedDurableState in extractCreationTime implementations akka/akka-projection#723

akka-projection-grpc added?

• akka/akka-projection@70aa953 (30k line merge)
• akka/akka-projection@4f051e5
• more changes following from this

[He-Pin]

The grpc projection is for s2s projection without a database been involved, which improve the performance, would be a nice addition.

[raboof]

I agree it's an interesting improvement, but I think we should probably let someone who actually wants to use it do the integration work, not work on it 'pre-emptively'.

[ptrdom]

I think the Akka gRPC projections depend on licensed dependencies provided by Akka - you need to add a custom resolver to sbt and register to get a token to be able to use it - so it will not be an option for Pekko.

It would be very nice to have a Pekko-based alternative, but that is definitely quite an undertaking to design and implement. I will likely work on something similar eventually, but HTTP/REST-based, mainly to enable more flexible encoding.

[raboof]

I think the Akka gRPC projections depend on licensed dependencies provided by Akka - you need to add a custom resolver to sbt and register to get a token to be able to use it - so it will not be an option for Pekko.

Ah I didn't realize - that's indeed likely not interesting for us. A non-persistent projection transport might be interesting even without the build tooling though.

It would be very nice to have a Pekko-based alternative, but that is definitely quite an undertaking to design and implement. I will likely work on something similar eventually, but HTTP/REST-based, mainly to enable more flexible encoding.

Interesting - gRPC seemed like a natural choice due to the first-class streaming, but curious what you come up with.

[He-Pin]

HTTP/REST-based is bad, we actually implement MCP transport with grpc and hsf at work, why can't we implement the same sbt plugin for grpc-projection?

[pjfanning]

https://mvnrepository.com/artifact/com.lightbend.akka/akka-projection-grpc_3/1.4.2 does not appear to have any dependencies on Akka libs that we won't be able to find Pekko equivalents of.

[ptrdom]

Interesting - gRPC seemed like a natural choice due to the first-class streaming, but curious what you come up with.

Since Akka/Pekko gRPC implementation is based on Akka HTTP, everything that allows it to provide first-class streaming is already present in HTTP, so the implementation could as well just provide a more generic HTTP/REST-enabled API instead of limiting the users to only gRPC, at least that is my thinking. I guess it is nice that gRPC has streams as a concept in the schema definition, but I do not see how it critically matters in the case of projections - users do not directly interact with the gRPC schema. Even in the case of HTTP/REST-enabled implementation, users should not directly interact with HTTP/REST protocol-level implementation details. Overall it would be nice to come up with a service-to-service communication algebra, then provide interpreters that users can choose based on their needs.

HTTP/REST-based is bad, we actually implement MCP transport with grpc and hsf at work, why can't we implement the same sbt plugin for grpc-projection?

Bad is a rather strong word, maybe it would be better to specify that it has non-ideal performance, but this is already a problem in Akka/Pekko gRPC - as far as I can see there is nothing special performance-wise in Akka/Pekko gRPC implementation, it uses Akka/Pekko HTTP HttpEntity.Chunked, so HTTP/1.1 chunked transfer encoding. Now if we wanted to improve that I am sure there are various options with their own pros/cons that an implementor can explore.

https://mvnrepository.com/artifact/com.lightbend.akka/akka-projection-grpc_3/1.4.2 does not appear to have any dependencies on Akka libs that we won't be able to find Pekko equivalents of.

Aha, I think I incorrectly thought that the dependency is on a library, where it actually is on codegen for sources, but we do seem to have codegen in Pekko too - https://pekko.apache.org/docs/pekko-grpc/current/buildtools/sbt.html. So we should be fine to implement gRPC projection for Pekko.

[He-Pin]

If you implement it with http2,then the performance should be great then.