The BSON serializer currently crashes for many inputs. I believe this is because the serialized format for a specific email may contain the \n character, e.g. in attachments... but we also use this character as a record separator to be able to read/write multiple serialized emails to a single file. This means that when we try to deserialize an email, reading to the first \n character may give us back a corrupt BSON record if the original email contained a raw \n character.
A suggested fix is either to find a way to stream multiple records in a single file pointer (similar to the CBOR implementation), or find another record delimiter which is guaranteed not to clash with any data in the file. But of course, other approaches that fix the problem are welcome too!
The BSON serializer currently crashes for many inputs. I believe this is because the serialized format for a specific email may contain the
\ncharacter, e.g. in attachments... but we also use this character as a record separator to be able to read/write multiple serialized emails to a single file. This means that when we try to deserialize an email, reading to the first\ncharacter may give us back a corrupt BSON record if the original email contained a raw\ncharacter.A suggested fix is either to find a way to stream multiple records in a single file pointer (similar to the CBOR implementation), or find another record delimiter which is guaranteed not to clash with any data in the file. But of course, other approaches that fix the problem are welcome too!