It appears that I was wrong about the simultaneous transfer speeds. The 250KB/s speeds up and down were miscalculated by the transfer program (seemed to have been showing a high average speed). Apparently the Business Ethernet (in this case it is an XO circuit provisioned by Speakeasy) only supports 3Mb total, not up AND down (for 6Mbit total). So if I am transferring up and down at the same time in theory I should only have 1.5Mbit up and down or 187.5KB/s at the maximum (if there was zero overhead).

Are you sure the bottleneck is your connection?

Do you also see this behavior when the simultaneous upload and download are occurring on different systems, or only when one system is handling both the upload and download?

If the problem goes away when independent machines are doing the work, the bottleneck is likely closer to the hard drive.

This sounds expected from my experience with lower end lines. On a home line, I ve found that traffic shaping and changing buffer sizes can be a huge help.

TCP/IP without any unusual traffic shaping will favor the most aggressive traffic at the expense of everything else. In your case, this means responses to the outgoing ACKs and such for the download will be delayed or maybe even dropped. See if your HN404 supports class based queuing or something similar and try it out.

Yes it is expected. This is symptomatic of any case in which you have a throttled or capped connection. If you saturate your uplink it will affect your downlink and vice versa.

This is because the your connection s rate-limiting impacts the TCP handshake acknowledgement packets (ACKs) and disrupts the normal "balance" of how these packets flow.

This is very thoroughly described on this page about Cable Modem Troubleshooting Tips, although it is not limited to cable modems:

If you saturate your cable modem s upload cap with an upload, the ACK packets of your download will have to queue up waiting for a gap between the congested upload data packets. So your ACKs will be delayed getting back to the remote download server, and it will therefore believe you are on a very slow link, and slow down the transmission of further data to you.

So how do you avoid this? The best way is to implement some sort of traffic-shaping or QoS (Quality of Service) on individual sessions to limit them to a maximum throughput based on a percentage of your total available bandwidth.

For example on my home network I have it so that no outbound connection can utilize any more than 67% (2/3rd) of my 192Kbps uplink. That means any single outbound session can only utilized 128Kbps, therefore protecting my downlink speed by preventing the uplink from becoming saturated.

In most cases you are able to perform this kind of traffic-shaping based on any available criteria such as source ip, destination ip, protocol, port, time of day, etc.