Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a race condition preventing embedded compiler to shutdown after a protocol error #2106

Merged
merged 3 commits into from
Oct 5, 2023
Merged

Fix a race condition preventing embedded compiler to shutdown after a protocol error #2106

merged 3 commits into from
Oct 5, 2023

Conversation

ntkme
Copy link
Contributor

@ntkme ntkme commented Oct 4, 2023

There is an extreme rare race condition during compiler shutdown after a protocol error is send from host callable here:

dispatcher.sendError(handleError(error, stackTrace));

A reproduction looks like this:

require 'sass-embedded'

def try
  puts yield
rescue StandardError => e
  puts e
end

fn = lambda do |_args|
  # this is an invalid calculation that will trigger a protocol error
  Sass::Value::Calculation.send(:new, 'foo', [Sass::Value::Number.new(1)])
end

try do
  Sass.compile_string('a {b: foo()}',
                      functions: { 'foo()': fn }).css
end

try do
  Sass.compile_string('a {b: c}').css
end

The fundamental problem here is that StreamSink#close() is async operation which can take time if the system is slow. This gives us a chance to send a second message before compiler has been shutdown.

The sequence of events looks like:

  1. Host send first request that triggers a protocol error in host callable response.
  2. Compiler send ProtocolError to host, however it forget to set _requestError = true in this case.
  3. Host send another request, and this request would be accepted because nothing has been cleaned up yet.
  4. Because _requestError is not set, the thrown ProtocolError bubbles up as a CompileFailure.
  5. Compiler send CompileFailure for the first request, and this would set _checkedOut = false on the isolate.
  6. Compiler send CompilerSuccess for the second request, and get Unhandled exception: Bad state: Shouldn't receive a message before being checked out.

The compiler would get stuck after the unhandled exception. The fix here is to make sure _requestError = true is set to that it does not send an extra CompilerFailure later which would reset isolate state at incorrect timing.

Slow sink close can be emulated with the following patch to make it easy to reproduce:

diff --git a/lib/src/embedded/isolate_dispatcher.dart b/lib/src/embedded/isolate_dispatcher.dart
index 044e2b9c..b58732f6 100644
--- a/lib/src/embedded/isolate_dispatcher.dart
+++ b/lib/src/embedded/isolate_dispatcher.dart
@@ -118,7 +118,9 @@ class IsolateDispatcher {
         // isolate, so we just send them as-is and close out the underlying
         // channel.
         sendError(compilationId, error);
-        _channel.sink.close();
+        Timer(Duration(milliseconds: 100), () =>
+          _channel.sink.close()
+        );
       } else {
         _handleError(error, stackTrace);
       }

Copy link
Contributor

@nex3 nex3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

CHANGELOG.md Outdated Show resolved Hide resolved
@nex3 nex3 merged commit 310904e into sass:main Oct 5, 2023
44 checks passed
@ntkme ntkme deleted the fix-shutdown-race-condition branch October 5, 2023 00:40
jgerigmeyer added a commit to oddbird/dart-sass that referenced this pull request Oct 12, 2023
* feature.color-4: (30 commits)
  Update for `lch()` tests (sass#2108)
  Add support for relative color syntax (sass#2112)
  Update for `oklab()` tests (sass#2094)
  Poke CI
  Cut a release (sass#2107)
  Implement first class mixins (sass#2073)
  Fix a race condition preventing embedded compiler to shutdown after a protocol error (sass#2106)
  Switch to the GitHub-hosted MacOS ARM64 runner (sass#2103)
  Update the version of Sass used by the website on release (sass#2102)
  Bump actions/checkout from 3 to 4 (sass#2088)
  Bump docker/setup-qemu-action from 2 to 3 (sass#2089)
  Implement support for the relative color syntax of CSS Color 5 (sass#2098)
  Rephrase errors for numbers that must be unitless or % (sass#2101)
  Forbid LLM contributions (sass#2100)
  Update protocol-version during embedded-host-node release (sass#2097)
  Deprecate Deprecation.calcInterp (sass#2096)
  Avoid useless allocations for interpolations without maps (sass#2095)
  Fix an error during embedded compiler shutdown (sass#2092)
  Cut a release (sass#2090)
  Expose the containing URL to importers under some circumstances (sass#2083)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants