With this specific application, we can import data, but there are limitations due to its 2000-era handling of XML files.

1. HTML forms uploading files have to…
   1. Have all the packets be received by the server.
   2. Process the file with the browser connection still open.
   3. The server has to tell the browser everything was received and is done.
2. All this has to happen as one action within a 5 minute window.

A better method would allow just uploading the files to a page. Background processes would monitor that location and process the files independent of the browser. Notifications can be sent to alert the user the processing is done.

Or… recognize the echo XML file because you took too long and prevent it from being loaded or remove the data.

I ended up figuring out that if we split the files at about 5,000 records, then it should take about half the 5 minute window. I am pleased that for most I have seen that is true and about one in ten take so much longer that if I had cheated and gone with closer to the 5 minutes, I would be deleting duplicates. (This last is because some files are 50MB and others 100MB.)

The grumbling of this post is that I am on the 25th of 58 files. This is tedious. I am lamenting not creating a curl script to do this part for me. Automation is perfect for things like this.