Topic: HTTP Protocol Core Functionality Doctor

[dejwa]

Hi guys, could someone help me please? I need a really advanced recommendation from HTTP protocol core functions expert.

I need to download the data as fast as possible from the slow server.. For e.g., around 1.000 requests (1kb each file). The issue is 50 more users could be downloading, they may also open large number of threads to the slow server. And I want this to be done as fast as possible.

I already have settings like number of download app servers, and number of threads per server implemented. But what will be the best settings to use? Or best way to make this?

Is it good to download data using 50 servers, and 5 threads each server, or maybe 2 servers and 200 threads.. Or something other?

Please help if you can, I will offer a good reward for the solution. I'm using a Python language, but I can implement a solution in any other language if required.

Target server is just a regular Apache2 server, with defaults.

Thank you very much for your help!

[James Blond]

Hi Dejwa,
there are several projects doing this

https://github.com/axel-download-accelerator/axel
https://github.com/maxogden/nugget
https://github.com/lavv17/lftp
https://github.com/huydx/hget
https://github.com/Code-Hex/pget
https://github.com/aria2/aria2

[dejwa]

Hi, thank you very much for your reply.

I've been using ARIA2 for a couple of years, I will take a look to the other apps.

The issue is my file size is only 1kb, so many of ARIA2 features will be useless. Like downloading single file using multiple threads.

Do you know maybe how the slow web server will act when there is 100 of users downloading using multiple threads? Please note server will be really slow in this scenario, who will have the advantage?

The user downloading using 50 servers (50 IP addresses) and small number of threads (like 5 ), or user downloading using 1 server (1 IP address) but high number of threads (like 200 or 400)

I think my answer relies in the core functionality of the web server software.

Thank you for your help,
Best

[dejwa]

One more question please...

Do you know the Apache2 webserver will act when I open 200 or 400 threads to it? At the moment when there are 100 of users downloading, and the server is busy - speed is really slow.

Will it disconnect majority of my threads, and lower it down to for e.g., 10?

If this is the scenario, how it will act when I open 5 threads from 50 servers (50 different IP addresses) to it?

Thank you one more time.

[James Blond]

Apache can handle a lot of concurrent requests. Especially static files. if you are under 10k concurrent downloads you are usually fine.

I wonder why you can't combine all those small files into one large compressed file and do a single download.

[dejwa]

Naah, I will try to make it work.

Thank you very much for your help and enjoy