Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

potential fix for #2238 #2243

Merged
merged 5 commits into from
Oct 18, 2016
Merged

Conversation

finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Oct 6, 2016

This PR fixes a race between destruction of a global Mutex as part of main thread exit and attempt to lock it on a background thread. (fix for issues #2238, #2137 and #2009)

Note to reviewer: Initial approach was to conditionally disable full destruction of Mutex, but Mark had a better idea which I implemented instead.

@bdash
Copy link
Contributor

bdash commented Oct 6, 2016

While this avoids destroying the underlying pthread_mutex, the Mutex itself is still destroyed so any access from a secondary thread will still result in undefined behavior. The lifetime of the Mutex itself is what needs to be extended. I'd guess the lifetimes of whatever data the mutex protects would also need to be extended, otherwise you'd trade an assertion failure when locking the mutex with a crash when accessing objects that have been destroyed.

@finnschiermer
Copy link
Contributor Author

This is tricky. The protected objects, which is two std::maps, are both swapped for empty maps during "destruction". The swap is not protected and may not be atomic, so this is far from perfect. But protecting it by taking the lock risks causing a deadlock in the destruction sequence, so we don't. wrt to the mutex itself, this is more of a theoretical point, isn't it? Are you proposing that we build a heap allocated singleton containing the mutex and what it protects and accesses it through a pointer in a global variable?

@bdash
Copy link
Contributor

bdash commented Oct 7, 2016

wrt to the mutex itself, this is more of a theoretical point, isn't it?

Accessing an object after its destructor has run results in undefined behavior. While it may happen to work, it's impossible to ensure that it works reliably on all platforms we support. I think we should avoid relying on such things without exceptionally good cause.

Are you proposing that we build a heap allocated singleton containing the mutex and what it protects and accesses it through a pointer in a global variable?

From what I can see an approach along those lines would address all of the problems in this area.

  • We'd no longer attempt to destroy a pthread_mutex that is in use.
  • We'd not risk undefined behavior by continuing to use a Mutex object after its destructor has run.
  • We'd eliminate the data races that exist due to the vector's being modified during static destruction while another thread may be accessing them (the current behavior seems like it could easily result in crashes).

Note that a singleton isn't strictly necessary as the variables in question could be independently heap-allocated, for instance. I don't think the heap allocation is strictly necessary either, but it's certainly the simplest approach.

@realm-ci
Copy link
Contributor

Please check your coverage here: https://ci.realm.io/job/realm/job/realm-core/job/PR-2243/2/Diff_Coverage

@finnschiermer finnschiermer changed the base branch from master to develop October 13, 2016 13:00
@realm-ci
Copy link
Contributor

Please check your coverage here: https://ci.realm.io/job/realm/job/realm-core/job/PR-2243/5/Diff_Coverage

std::map<std::string, std::weak_ptr<SlabAlloc::MappedFile>> all_files;
util::Mutex all_files_mutex;
// prevent destruction at exit (which can lead to races if other threads are still running)
std::map<std::string, std::weak_ptr<SlabAlloc::MappedFile>>& all_files =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These declarations could be good places to use auto as it would avoid repeating the type name.

// prevent destruction at exit (which can lead to races if other threads are still running)
util::Mutex& mapping_mutex = *new Mutex;
std::vector<mapping_and_addr>& mappings_by_addr = *new std::vector<mapping_and_addr>;
std::vector<mappings_for_file>& mappings_by_file = *new std::vector<mappings_for_file>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this cause valgrind leak reports?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've verified it doesn't - same goes for the address sanitizer. Would be logical if it did, though.

}
} at_exit;
// prevent destruction at exit (which can lead to races if other threads are still running)
util::Mutex& mapping_mutex = *new Mutex;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just remove pthread_mutex_destroy() in ~Mutex()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that should follow from the discussions above.

@rrrlasse
Copy link
Contributor

seems to do the job.. 👍

@finnschiermer finnschiermer merged commit fd54f5e into develop Oct 18, 2016
@finnschiermer finnschiermer deleted the fsa/fix_mutex_destruction_race branch October 18, 2016 14:46
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants