Design Decisions

[billroper]

The problem with coming to a project at a later stage of development is that a number of design decisions will have been made for you. Right now, I'm discovering that some of them seem to be fairly marginal.

We are attempting to move some large binary data objects (measured in tens of megabytes) across an HTTP connection in XML format. While this seems to work when we move the files one at a time, when we try to move multiple files in sequence, I start to see failures. Lots of failures.

After struggling with this and spending a lot of time watching the memory allocation in the receiving application, I have discovered the following:

We're parsing the XML with MSXML 6's DOM parser. This does its own garbage collection. Someday. Maybe never, at least for the purposes of this application. The result is that we never seem to get the memory back after freeing up one of the objects.

The binary data is being passed in a series of CDATA sections. This does not appear to be a bulletproof method of doing this. Apparently, Base64 encoding would be better, if I only had a handy Base64 encode/decode app that I was sure would work. There's one in Boost, but it may not be quite right, according to a bug report I found as I was chasing down the topic on Google.

We had a big fat memory leak of our own due to mishandling BSTRs. I've fixed that one.

I think I will sleep on this. Tonight. And tomorrow night. And Sunday night...