How did I struggle with the task scheduler service WebSitePanel

We received an application from the user that they stopped working in WebSitePanel scheduled tasks. When stopped working is not known. Most likely, when we updated the panel to the new version (from 2.0 to 2.1). In the new version of the panel, the developers decided to bring the Task Scheduler service to a separate Windows service. Previously, it was part of the Enterprise server.

I think this service probably flew out, I go to the server and look, indeed the service is stopped. Running. I think everything is OK, I'm preparing to unsubscribe to the client, I'm watching the processes - there is no service, I update the snap services - the service is stopped. For the sake of truth I try to run it again, the same result.

I go, see what is written by viewing the events, there are two events, the first that the service was unexpectedly completed, the second that:

Application: WebsitePanel.SchedulerService.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.NullReferenceException

Stack:

at WebsitePanel.EnterpriseServer.Scheduler.RunBackgroundTask(WebsitePanel.EnterpriseServer.BackgroundTask)

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

at System.Threading.ThreadHelper.ThreadStart()

Very informative ... I'm climbing the forums, they recommend comparing the configuration files of the Enterprise server (web.config) and the configuration files of the designer - WebsitePanel.SchedulerService.exe.conf, they should have the same connection lines and encryption keys.

СI'm equal, there are no encryption keys in the config file. Satisfied as an elephant, I insert a key, I save config, I go into services, I launch, I check. Horseradish!!! The service is stopped. In the logs, nothing has changed.

Another recommend to compare the versions of dlls, compare, all the same.

Well, I think this scheduller can be reinstalled? I start the installer, he does not see it, he suggests to install it again. A little nervous, no matter how the installer did not crash Enterprise Server (because it should not be active in the installer, that would not overwrite the files), I decided to install, manually deleting the service, and all the scheduler files. No result.

I assume that it is in .net. The server is 4.5, in configurations, there are always 4.0. Played with the configs, trying to change the version in them. The service can not start up, swears at an unknown version of .net.

Okay. I assume that you need to install a clean version of .net 4, delete .net 4.5, at the same time I install all updates to the server, they accumulated about 70 pieces. I restart the server, rubbing my hands, that updates and changing the version of .net will help ... The same.

While I thought what to do next, I created a couple of test problems, and started to run the service. And I found that one of the tasks successfully completed once, out of 10 runs of the task and the service. Hmm...

Ok, I think it's time to see what happens in the database, I go to the database on the server, and there...

...The devil will break his leg from the number of tables, looked at the names, where the Schedule appears, looked at what is in the tables, did not see anything like that. I saw that all the tasks are stored in the Schedule table, and there ID space hosting is indicated to whom these tasks belong. Here you can turn them off by changing the column Enabled.

I think need to try to disconnect all tasks and see if the service will fall out. I've gone to the page of the panel, in it I switch to hosting the space by its ID, I've disabled the task. To make sure that it is in this table that you make a change to enable / disable the job. Yes, the change was made. I save the current state of the table to a file, and sql command "Update Schedule set Enabled = 0 Where Enabled = 1" disables all tasks.

Taking an opportunity, I looked, that would be hosting space, in the assignments existed.

Thinking that there can not be any more problems with SQL, I'm going to the server with the shadeler. I start the service, the hell out there ... The error is the same ...

OK, WebSitePanel is an open source project, I download sources from the githab, I find in the project the code for the Scheduler service. In it I find where the method RunBackgroundTask () is caused, I add on it the exception handler, I compile, I copy new dlls on the server, I start service, the service is stopped. But the error is different:

Application: WebsitePanel.SchedulerService.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.NullReferenceException

at WebsitePanel.EnterpriseServer.Scheduler.RunBackgroundTask(WebsitePanel.EnterpriseServer.BackgroundTask)

at WebsitePanel.EnterpriseServer.Scheduler+<>c__DisplayClass3.<RunManualTasks>b__1()

at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

at System.Threading.ThreadHelper.ThreadStart()

I find where the RunManualTasks method is called in the code, and I add an exception handler to this hostel. I compile, the service does not crash, but neither the fig nor any jobs are working ...

I start looking for what these methods do, it turns out that they call stored procedures on the SQL server. I suppose that maybe there is not enough stored procedure, compare them to the working sql and test, where the scheduler works. By number - the same. By content you'll be smart enough to check, there are 450 of them.

I decide to start updating the database, which was started when the database was upgraded to 2.1, and to look at the errors, there are no serious errors, and they are not related to stored procedures, so stored procedures should be in order, as well. They are updated with this update.

I understand that you need to understand which query, and what response comes from the server's sql server. At this point, I did not have full rights to sql, not for the need, but the person who could give them away. In order not to lose time, I run Wireshark on the server where the vendor is, and I see that the latter calls the stored procedure - GetProcessBackgroundTasks, but the answer does not come to it, the error message is more accurate.

Okay, I run this stored procedure, it returns something, I go into it, to see what tables it takes, it is quite logical to take the tasks from the table backgroundtasks. I see that in this table 3 tasks. I suspect that one of them is ruining the service.

By this point, the man who gave me full SQL rights just came up. I run sql profiler, and I remove the request queue, I see that the service actually falls when exactly one of these jobs is returned.

In the job ID was specified for the hosting of the space he created and called. Compared with the table hosting space, and hosting space is not so ... So at first, when I looked at the scheduler tables, I did not suspect that it could be wrong with the database. Because For all hosting space I turned off the job, and neither could not assume that the job may hang in the database, after the user is deleted.

In general, I deleted this line from the table backgroundtasks, it was dependent on one more line in another table - BackgroundTaskStack. After that the service successfully started. Apparently when deleting a user from the panel, something went wrong and left his hung task. Why this happened is a mystery.

Here such here, labor everyday life. =)