Buffering Bug
A bug that can be observed to exist and one can believe that some setting or thing can be changed but 2 weeks of attempting to use every variation of command structure, and detailed analysis of possible causes goes nowhere certainly makes one consider other options.
If you stick with debugging for too long and the client needs a result it makes you think harder. The first serve worked fine, there was little difference with the second POST but it nearly always broke the XML result. It does not seem at all logical so the programmer gets it in the neck.
One such event that would not debug, after 10 days of infuriating and unjust buffer corruption investigations, I suddenly saw what was happening. The PHP system command was running for too long, and returned with a broken buffer. But only on the second load, never the first. We spent days looking for bugs in the XML, trying every option under the sun. Then it suddenly hit me and I spent 3 hours devising a method that would always work, and what remained was some front end browser scripting to keep the user satisfied things are happening while their data was arriving.
In a multitasking environment, there are two things to worry about when serving data from the same query. The server may queue the query and take time to supply the data, and it is more likely to do so if it is still channeling out data from the previous query. Not that that is a logical cause of the problem, but it may create an issue that excites the problem so that PHP thinks the second system call has had long enough and carries on with half an XML delivery. Half an XML is nothing.
It can not be fixed with the “right settings”.
When you offload a buffer to a file it absorbs it - in one gulp. When you offload a buffer to a browser, it gets chopped into TCP segments which are delivered less quickly. As soon as a large delivery has bits of time added to it, it changes the delivery rules.
The problem is that the process that is held up by the slow direct to browser delivery so the OS reduces the IO priority for that process. What you must be able to do with a content stream that is potentially megabytes in one delivery is send it as fast as the line can take it. But not after a chopped delivery.
If you send it slower – by latching the browser process to the file creation process, if there is any significant quantity of data, you are telling the operating system that your data is slow and therefore to give it the priority and rules of a slower line. Your next query on that channel is going to contend with the results the first query established and tell routers and servers along the way.
There is a reason for multi tasking technology. One of them is the disparity between server buffers, disks and the bottlenecks of TCP traffic serving.
A better way would be to stream the data through multiple UDP connections, but that is a 10k solution. An even better one is to use P2P but there are a few billion dollar company to compete with who are spending tens of millions of such technology for rapid streaming delivery.
The most effective easy, safe and inexpensive way to multitask with large files is to:
a) safely and atomically isolate the capture in the most efficient manner
b) deliver the datastream to the line without any delay states (so direct serving while processing is extremely vulnerable to buffer breakage).
