Log Connection ID in the Jobs table

[ShuangLiBCM]

Currently, checking the jobs of a table only returns the initial timestamp. It will not return current status of the job. This will create issue when we tries to terminate the truly dead jobs, as killing undergoing jobs will sometimes create unhealthy termination of the running program when it tries to update the table and find that the job list is no longer there. One possible solution is to create a heartbeat, which will periodically return the status of the job.

[dimitri-yatsenko]

Indeed, if a make_tuples call gets terminated abruptly (without the normal chance to raise an exception) while populating, then its job reservation record remains with the reserved status. So there is no way to tell a dead job from a long one.

The solution of periodically updating the jobs entry is not simple because the connection is in an isolated transaction during a make_tuples call, so any updates to the jobs table will not be seen by other transactions until make_tuples exits.

Instead, I propose that the job table should store the connection id. And the Jobs object will display which of the connections are still alive.

[eywalker]

That's a brilliant solution!