Don't use BinaryFormatter for data you are serializing to put over the network - there is a lot of extra cruft it adds that you probably do not need[1]. Use a MemoryStream and a BinaryWriter, then the byte array you get out of that, to put over the network.
[1] - unless you need the deep copy of the graph and the references and all it does give you.