How to safely perform rolling updates on an Akka Cluster

How to safely perform rolling updates on an Akka Cluster

While trying to upgrade an Akka Cluster you may receive a message akin to:

Failed to deserialize message from [akka://SYSTEM_DETAILS] with serializer id [6] and manifest []. java.lang.IllegalArgumentException: Unknown manifest [IJ]

This warning in particular means that serialization logic (identified by the serializer manifest) was added in some version in between an older Akka version and the newer one you are trying to upgrade, so that when a message is serialized and sent to the old node it does not know how to deserialize it and therefore drops it.

That noted, we maintain wire compatibility to support rolling upgrades but it can often require stepwise upgrades over patch versions. The notes around that (especially version compatibility) are in the docs here. Ultimately, if you are looking for a rolling upgrade you will need to do this in steps, allowing for compatibility to increment one incompatibility at a time.