Remove Un-necessary tries to mark execution complete when it has already been cancelled #4199

udsamani · 2024-07-03T09:53:28Z

This PR aims at the following

Make sure we do no try to mark the execution as Completed if the user has already cancelled it.

coderabbitai · 2024-07-03T09:53:43Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

wdbaruni · 2024-07-03T18:25:20Z

I have realized bunch of other issues with our execution cancellation logic that are more serious. It seems we have a race condition where we can mark an execution as completed instead of cancelled.

Inside endpoint, we trigger the execution cancellation first, which is then carried in a go routine, and then update the state of the execution to cancelled:

bacalhau/pkg/compute/endpoint.go

Lines 143 to 154 in a766d1c

    
           if localExecutionState.State.IsExecuting() { 
        
           	err = s.executor.Cancel(ctx, localExecutionState) 
        
           	if err != nil { 
        
           		return CancelExecutionResponse{}, err 
        
           	} 
        
           } 
        
           err = s.executionStore.UpdateExecutionState(ctx, store.UpdateExecutionStateRequest{ 
        
           	ExecutionID: request.ExecutionID, 
        
           	NewState:    store.ExecutionStateCancelled, 
        
           	Comment:     "execution canceled due to: " + request.Justification, 
        
           })

bacalhau/pkg/compute/executor_buffer.go

Lines 201 to 216 in a766d1c

    
           go func() { 
        
           	ctx := logger.ContextWithNodeIDLogger(context.Background(), s.ID) 
        
           	ctx = system.AddJobIDToBaggage(ctx, execution.Job.ID) 
        
           	ctx = system.AddNodeIDToBaggage(ctx, s.ID) 
        
           	ctx, span := system.NewSpan(ctx, system.GetTracer(), "pkg/compute.ExecutorBuffer.Cancel") 
        
           	defer span.End() 
        
           	err := s.delegateService.Cancel(ctx, localExecutionState) 
        
           	if err == nil { 
        
           		s.mu.Lock() 
        
           		defer s.mu.Unlock() 
        
           		delete(s.running, execution.ID) 
        
           	} 
        
           }() 
        
           return nil

If executor cancellation was fast, then it will actually mark it as completed instead of cancelled. I was able to reproduce this by injecting a sleep

What we should do is, first update the execution state, and then cancel the executor.

In addition to that, e.Wait doesn't return an error or any indication that it was interrupted or cancelled, which lets the executor to think it was successful and then attempts to publish results and mark it as completed successfully. We should also make Wait return an error if the execution didn't complete all the way

bacalhau/pkg/compute/executor.go

Lines 336 to 338 in a766d1c

    
           result, err := e.Wait(ctx, state) 
        
           if err != nil { 
        
           	if errors.Is(err, context.DeadlineExceeded) {

…when-cancelling-job

wdbaruni

LGTM. Just minor comments. How difficult to add a test case to verify the changes work as expected?

wdbaruni · 2024-07-04T10:25:14Z

pkg/compute/executor.go

@@ -300,7 +302,7 @@ func (e *BaseExecutor) Run(ctx context.Context, state store.LocalExecutionState)
 	stopwatch := telemetry.Timer(ctx, jobDurationMilliseconds, state.Execution.Job.MetricAttributes()...)
 	topic := EventTopicExecutionRunning
 	defer func() {
-		if err != nil {
+		if err.Error() != executor.ErrAlreadyCancelled.Error() {


better to check using !errors.Is(err, executor.ErrAlreadyCancelled)

wdbaruni · 2024-07-04T10:27:00Z

pkg/compute/executor.go

+	// There can be a scenario where the execution may already have been cancelled
+	// by the user. In such case we should not mark it as completed.
+	currentExecutionState, err := e.store.GetExecution(ctx, execution.ID)
+	if err != nil {
+		return err
+	}
+	if currentExecutionState.State == store.ExecutionStateCancelled {
+		log.Ctx(ctx).Debug().Msgf("Execution has already been cancelled for %s", execution.ID)
+		return nil
+	}
+


do we still need this check since cancelling the execution would trigger an ErrAlreadyCancelled

udsamani · 2024-07-04T11:50:56Z

LGTM. Just minor comments. How difficult to add a test case to verify the changes work as expected?

Let me take a stab at adding tests :)

fix misleading logs on already cancelled jobs

4267495

udsamani requested review from wdbaruni and frrist July 3, 2024 09:53

make the log debug

7030b0c

udsamani self-assigned this Jul 3, 2024

udsamani added 2 commits July 4, 2024 09:21

Merge remote-tracking branch 'origin/main' into 3698-incorrect-state-…

c242c5c

…when-cancelling-job

a potential way of cancelling correctly

b90dbc0

wdbaruni reviewed Jul 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Un-necessary tries to mark execution complete when it has already been cancelled #4199

Remove Un-necessary tries to mark execution complete when it has already been cancelled #4199

udsamani commented Jul 3, 2024

coderabbitai bot commented Jul 3, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

wdbaruni commented Jul 3, 2024

wdbaruni left a comment

wdbaruni Jul 4, 2024

wdbaruni Jul 4, 2024

udsamani commented Jul 4, 2024

Remove Un-necessary tries to mark execution complete when it has already been cancelled #4199

Are you sure you want to change the base?

Remove Un-necessary tries to mark execution complete when it has already been cancelled #4199

Conversation

udsamani commented Jul 3, 2024

coderabbitai bot commented Jul 3, 2024

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

wdbaruni commented Jul 3, 2024

wdbaruni left a comment

Choose a reason for hiding this comment

wdbaruni Jul 4, 2024

Choose a reason for hiding this comment

wdbaruni Jul 4, 2024

Choose a reason for hiding this comment

udsamani commented Jul 4, 2024

CodeRabbit Configration File (`.coderabbit.yaml`)