JobScheduler MultiTenant | appropriate Object Space to the AdditionalObjectSpaces

[devtka]

Good day All,

JobScheduler not working in MultiTenant?

i added in the 2 Projects:

• Xpand.XAF.Web.All
• Xpand.XAF.Modules.JobScheduler.Hangfire
• Hangfire 1.7.35
• Hangfire.AspNetCore
• Hangfire.Core
• Hangfire.SqlServer
• in BlazorModule
  RequiredModuleTypes.Add(typeof(Xpand.XAF.Modules.Blazor.BlazorModule));
  RequiredModuleTypes.Add(typeof(Xpand.XAF.Modules.JobScheduler.Hangfire.JobSchedulerModule));
  RequiredModuleTypes.Add(typeof(Xpand.XAF.Modules.Email.EmailModule));
• in Startup.cs
  [assembly: HostingStartup(typeof(HostingStartup))] [assembly: HostingStartup(typeof(HangfireStartup))] [assembly: HostingStartup(typeof(Xpand.XAF.Modules.Blazor.BlazorStartup))]
  and:
  // Added: HangFire Service var hangConnectionString = Configuration.GetConnectionString("HangFireConnectionString"); services.AddHangfire(configuration => configuration .UseSimpleAssemblyNameTypeSerializer() .UseRecommendedSerializerSettings() .UseSqlServerStorage(hangConnectionString) ); services.AddHangfireServer();

1. Project without Tenant
no errors, working

2. Project with Tenant
System.ArgumentException: "Cannot handle the "Xpand.XAF.Modules.JobScheduler.Hangfire.BusinessObjects.CronExpression" type. To address this error, add an appropriate Object Space to the AdditionalObjectSpaces collection (see http://docs.devexpress.com/eXpressAppFramework/DevExpress.ExpressApp.CompositeObjectSpace.AdditionalObjectSpaces)."

Project with Tenant:
DemoJobSchedulerTenant (2).zip

[yunits]

One thing that I would check is your connection string for hangfire.

I guess you cannot use the database from tenant super admin:

"ConnectionString": "Integrated Security=SSPI;Pooling=false;Data Source=(localdb)\\mssqllocaldb;Initial Catalog=DemoJobSchedulerTenant_Service"
"HangFireConnectionString": "Integrated Security=SSPI;Pooling=false;Data Source=(localdb)\\mssqllocaldb;Initial Catalog=DemoJobSchedulerTenant_Service"

Edit: I also got this error message above with another business object when I was about to modify unintentionally the _Service database and created business objects there.

[yunits]

I played around a bit and came to the conclusion that there is more needed than modified connection string.

apo is the pro in case of multi tenancy, therefore lets wait for his answer.

For example do we need multiple hangfire server/nodes for every tenant? or is it possible to configure one hangfire server for multiple queues and restrict hangfire dashboard to specific queue from tenant?
https://discuss.hangfire.io/t/advice-on-setting-up-hangfire-for-multi-tenant-system/7672
https://stackoverflow.com/questions/57394712/hangfire-multi-tenant-asp-net-core-resolving-the-correct-tenant

[apobekiaris]

I am sorry there are no tests or any kind of effort to this case. Whatever I will say it maybe totally missleading but I in any case the tenant owns its data so it should own the hangfire node

For example on the first look (maybe totally wrong) I do not agree with SO anwer, Hangfire knows nothing about Xaf passing a tenantId argument around is incomplete, there are much more to consider.

This is a complex case that requires many resources and needs to consider that actual bussiness case, meaning a common node implies that it has access to all tenant data at some level at least, is that ok? as always it depends

[yunits]

I would suggest to modify less as possible, therefore hangfire gets its own connection string, neither host nor tenant, every table in this database gets a column "tenant_id" and every view gets filtered by criteria tenant_id.

If tenant_id is NULL, it should also work with non tenant versions.

Dashboard/node would be step 2 after this modification but it would be nice to have first hangfire working in tenant environment at least with one node.

Multiple nodes could be seen as an enhancement.

[yunits]

maybe someone of you will find time to test this or another simple solution.

I can test at earliest on next week.

[apobekiaris]

I spent some time on the case, but I do not have a solution to it yet. Need to consult the Xaf team as hangfire is a rather complex module. In short however my impression is that will be one hangfire node and every tenant can push his own jobs, Xaf is responsible to inject the correct IServiceProvider for each job. I already fixed the reported problem but currently Xaf injects the super admin provider which is not what we want.

[yunits]

is there some way to test your current progress?

[apobekiaris]

but currently Xaf injects the super admin provider which is not what we want.

i am not sure what u want to test unless I spend enough time to understand how to actually resolve the correct IServiceProvider u will only be able to use handfire with the super admins BO so not of much value. Unless u go into trouble creating all from scratch Datalayers etc which again is not deserve your time

[yunits]

I understood the actual problem halfway.
I think this is beyond my expertise, but I just wanted to take a look. Maybe you're right and I should wait.

You can't resolve the correct IServiceProvider, for example unless the tenant has logged in. And after login you would need to "refresh" hangfire I guess, otherwise SuperAdmin is still in use, as hangfire loads everything at blazor startup.
Also hangfire needs to be able to work in background without any login.

Therefore I thought: wouldn't it be possible to create a user "hangfire" for every tenant and then for example create an ObjectSpace with login credentials and get IServiceProvider from ObjectSpace.ServiceProvider?
Something like that.

[apobekiaris]

for example unless the tenant has logged in

really great catch have to experement on this cause we already support the SecuredObjectSpace so it might work I just did the wrong tests. Will post the fix for u experement more u sound genious!

[apobekiaris]

The pre-release 4.241.3.0 in the Reactive.XAF lab branch includes commits that relate to this task:

• [JobScheduler MultiTenant | appropriate Object Space to the AdditionalObjectSpaces

#1065](eXpandFramework/Reactive.XAF@e3a62e4)

To minimize version conflicts we recommend that you use the Xpand.XAF.Core.All, Xpand.XAF.Win.All, Xpand.XAF.Web.All packages. Doing so, all packages will be at your disposal and .NET will add a dependecy only to those packages that you actually use and not to all (see the Modules installation-registrations youtube video).

Released packages:

No packages released.

Please update the related Nuget packages and test if issues is addressed. These are nightly nuget packages available only from our NugetServer.

If you do not use these packages directly but through a module of the main eXpandFramework project, please wait for the bot to notify you again when integration is finished or update the related packages manually.

Thanks a lot for your contribution.

[yunits]

I am glad that I could help you somehow.

The initial error has been solved. Now I get the same error with another BO:
"Cannot handle the "Xpand.XAF.Modules.JobScheduler.Hangfire.BusinessObjects.Job" type."

I've created a sample with a Testjob
Hangfire.zip

Steps to reproduce:

1. run Blazor App
2. login as admin
3. log off and restart Blazor App
4. login with admin@company1.com
5. create a new job and trigger

@apobekiaris

[apobekiaris]

sorry for not answering, thnks for the sample it looks like the same issue which unfortunately currently I do not have the resources to spent. Meaning the Job is not recognized cause its the super admin application and not the tenant. The way this can be solved as I mentioned above is to force a tenant login from the job. There are many methods in this framework that may help but I have not tested anything. Also there is this test where the SecuredObjectSpaceProvider is tested so demonstrates a possible approach to user logon from a job.

https://github.com/eXpandFramework/Reactive.XAF/blob/7575776133d75b849e46bc2297e83b69304af75e/src/Tests/JobScheduler.Hangfire/JobSchedulerTests.cs#L78

[yunits]

I've tried to change the Application.ConnectionString from SuperAdmin ConnectionString to Tenant ConnectionString to have access to the Tenant database, but that didn't work, I get the same error message but the connection should be the right one.

here has someone done a similar task: https://www.youtube.com/watch?v=tc0Q3Jcy1oE&t=234s

If this would work, I would only need tenant_id in the Jobs Table and could change ConnectionString.
I will wait until you have some time to spent on this issue again, I'm not getting ahead.

[yunits]

T1254662