I recently had some pleasant experience with ZeroMQ, http://www.zeromq.org/
I do not have personal experience with MPI, though my associates have.
The only thing I know about ZeroMQ's internals when making a cmake build setup was that it uses C++ though its interface is C with opaque pointers and has status code functions.
ZeroMQ has in process, inter-process, and normal networked TCP communications as well as pragmatic multicasting with the same interface, how likely is it to have at least the first two on epiphany cores?