Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreaded treated as single thread #15

Open
openbullet opened this issue Dec 27, 2019 · 2 comments
Open

Multithreaded treated as single thread #15

openbullet opened this issue Dec 27, 2019 · 2 comments

Comments

@openbullet
Copy link

openbullet commented Dec 27, 2019

Hello,
first of all I'd like to thank you for your amazing library!

I want to solve multiple challenges at the same time in a multithreaded way.

In this snippet I will execute a get request in parallel for the same website 5 times, using the standard HttpClient without the solver:

Stopwatch sw = new Stopwatch();
var jobs = 5;
var target = new Uri("http://example.com");

var tasks = Enumerable.Range(1, jobs).Select(async n =>
{
    using (var handler = new HttpClientHandler())
    {
        using (var client = new HttpClient(handler))
        {
            await client.GetAsync(target);
            Console.WriteLine($"[{sw.Elapsed.TotalSeconds} s] [Job {n}] Completed");
        }
    }
});

sw.Start();
Console.WriteLine($"Started {jobs} jobs");
Task.WhenAll(tasks).Wait();
sw.Stop();

The output will look like this:

[0,481979 s] [Job 4] SUCCESS
[0,4822557 s] [Job 5] SUCCESS
[0,4830797 s] [Job 1] SUCCESS
[0,5180153 s] [Job 2] SUCCESS
[0,5286887 s] [Job 3] SUCCESS

So the jobs are completed correctly in no specific order and in around the same time, which is exactly what I want.

Now if I introduce the CloudflareSolver inside those tasks, I will write something like this

Stopwatch sw = new Stopwatch();
var jobs = 5;
var target = new Uri("http://example.com");

var tasks = Enumerable.Range(1, jobs).Select(async n =>
{
    using (var handler = new HttpClientHandler())
    {
        using (var client = new HttpClient(handler))
        {
            var cf = new CloudflareSolver() { MaxTries = 3, ClearanceDelay = 3000 };
            var result = await cf.Solve(client, handler, target);
            Console.WriteLine($"[{sw.Elapsed.TotalSeconds} s] [Job {n}] Completed");
        }
    }
});

sw.Start();
Console.WriteLine($"Started {jobs} jobs");
Task.WhenAll(tasks).Wait();
sw.Stop();

which for any website (both protected by cloudflare and not) gives this result (in the cloudflare case since it takes more time the timestamps are like 3s, 6s, 9s etc)

[0,6562882 s] [Job 1] SUCCESS
[0,8766025 s] [Job 2] SUCCESS
[1,1023446 s] [Job 3] SUCCESS
[1,3321279 s] [Job 4] SUCCESS
[1,5533806 s] [Job 5] SUCCESS

This is clearly a synchronous execution where only one solver is able to run at the same time even if I have multiple separate instances of the solvers.

Why do you think this happens?

Thank you for your help,
Ruri

P.S. the same happens even if I give it the ClearanceHandler directly.

@openbullet
Copy link
Author

In case you are wondering why I need to do this, instead of just solving the challenge once and then using the cookies for my multiple requests, it's because I want to use a different proxy for each of the 5 http clients, so I need to solve 5 different challenges since one challenge is valid only for a certain IP, as I'm sure you're already aware.

@openbullet
Copy link
Author

Hi, I fixed this issue by removing the SemaphoreLocker in CloudflareDetector.cs and removing the static keyword from the one in CloudflareSolver.cs
Do you want me to make a pull request or would you prefer to keep the lock? It does break multithread use so it's up to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant