主队列上的dispatch_sync和dispatch_async

Bear with me, this is going to take some explaining. I have a function that looks like the one below.

忍耐一下，这需要一些解释。我有一个像下面这样的函数。

Context: "aProject" is a Core Data entity named LPProject with an array named 'memberFiles' that contains instances of another Core Data entity called LPFile. Each LPFile represents a file on disk and what we want to do is open each of those files and parse its text, looking for @import statements that point to OTHER files. If we find @import statements, we want to locate the file they point to and then 'link' that file to this one by adding a relationship to the core data entity that represents the first file. Since all of that can take some time on large files, we'll do it off the main thread using GCD.

上下文:“aProject”是一个名为LPProject的核心数据实体，它包含一个名为“memberFiles”的数组，该数组包含另一个名为LPFile的核心数据实体的实例。每个LPFile表示磁盘上的一个文件，我们要做的是打开每个文件并解析其文本，查找指向其他文件的@import语句。如果我们找到@import语句，我们希望找到它们指向的文件，然后通过向表示第一个文件的核心数据实体添加关系，将该文件链接到这个文件。由于这在大型文件上都需要花费一些时间，所以我们将使用GCD在主线程上完成。

- (void) establishImportLinksForFilesInProject:(LPProject *)aProject {
    dispatch_queue_t taskQ = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
     for (LPFile *fileToCheck in aProject.memberFiles) {
         if (//Some condition is met) {
            dispatch_async(taskQ, ^{
                // Here, we do the scanning for @import statements. 
                // When we find a valid one, we put the whole path to the imported file into an array called 'verifiedImports'. 

                // go back to the main thread and update the model (Core Data is not thread-safe.)
                dispatch_sync(dispatch_get_main_queue(), ^{

                    NSLog(@"Got to main thread.");

                    for (NSString *import in verifiedImports) {  
                            // Add the relationship to Core Data LPFile entity.
                    }
                });//end block
            });//end block
        }
    }
}

Now, here's where things get weird:

现在，事情变得诡异起来:

This code works, but I'm seeing an odd problem. If I run it on an LPProject that has a few files (about 20), it runs perfectly. However, if I run it on an LPProject that has more files (say, 60-70), it does NOT run correctly. We never get back to the main thread, the NSLog(@"got to main thread"); never appears and the app hangs. BUT, (and this is where things get REALLY weird) --- if I run the code on the small project FIRST and THEN run it on the large project, everything works perfectly. It's ONLY when I run the code on the large project first that the trouble shows up.

这段代码有效，但我发现了一个奇怪的问题。如果我在一个有几个文件(大约20个)的LPProject上运行它，它会运行得很好。但是，如果我在有更多文件的LPProject上运行它(比如，60-70)，它就不能正确运行。我们永远不会回到主线程，NSLog(@“到达主线程”);永远不会出现，应用程序挂起。但是，(这就是事情变得非常奇怪的地方)——如果我先在小项目上运行代码，然后在大项目上运行它，那么所有的东西都可以完美地工作。只有当我首先在大型项目上运行代码时，才会出现问题。

And here's the kicker, if I change the second dispatch line to this:

这就是问题所在，如果我把第二个调度线换成这个

dispatch_async(dispatch_get_main_queue(), ^{

(That is, use async instead of sync to dispatch the block to the main queue), everything works all the time. Perfectly. Regardless of the number of files in a project!

(也就是说，使用async而不是sync来将block发送到主队列)，所有的工作都是有效的。完美。不管项目中文件的数量!

I'm at a loss to explain this behavior. Any help or tips on what to test next would be appreciated.

我无法解释这种行为。如果有任何关于下一步测试内容的帮助或建议，我们将非常感激。

3 个解决方案

#1

This is a common issue related to disk I/O and GCD. Basically, GCD is probably spawning one thread for each file, and at a certain point you've got too many threads for the system to service in a reasonable amount of time.

这是与磁盘I/O和GCD相关的常见问题。基本上，GCD可能为每个文件生成一个线程，并且在某个点上，您在合理的时间内为系统提供了太多的线程。

Every time you call dispatch_async() and in that block you attempt to to any I/O (for example, it looks like you're reading some files here), it's likely that the thread in which that block of code is executing will block (get paused by the OS) while it waits for the data to be read from the filesystem. The way GCD works is such that when it sees that one of its worker threads is blocked on I/O and you're still asking it to do more work concurrently, it'll just spawn a new worker thread. Thus if you try to open 50 files on a concurrent queue, it's likely that you'll end up causing GCD to spawn ~50 threads.

每次调用设置()和阻止你尝试任何I / O(例如,它看起来像你阅读一些文件),很有可能的线程执行的代码块是将块(被操作系统暂停),而它等待从文件系统读取数据。GCD的工作方式是这样的:当它看到它的一个工作线程在I/O上被阻塞，而您仍然要求它并发执行更多的工作时，它将生成一个新的工作线程。因此，如果尝试在并发队列上打开50个文件，很可能会导致GCD产生~50个线程。

This is too many threads for the system to meaningfully service, and you end up starving your main thread for CPU.

对于系统来说，这是太多的线程，无法提供有意义的服务，最终您会将主线程压缩为CPU。

The way to fix this is to use a serial queue instead of a concurrent queue to do your file-based operations. It's easy to do. You'll want to create a serial queue and store it as an ivar in your object so you don't end up creating multiple serial queues. So remove this call:

解决这个问题的方法是使用串行队列而不是并发队列来执行基于文件的操作。很容易做的。您将希望创建一个串行队列，并将其作为ivar存储在对象中，这样就不会最终创建多个串行队列。所以把这个叫:

dispatch_queue_t taskQ = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);

dispatch_queue_t taskQ = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);

Add this in your init method:

在你的init方法中添加这个:

taskQ = dispatch_queue_create("com.yourcompany.yourMeaningfulLabel", DISPATCH_QUEUE_SERIAL);

taskQ = dispatch_queue_create(“com.yourcompany。yourMeaningfulLabel”,DISPATCH_QUEUE_SERIAL);

Add this in your dealloc method:

在dealloc方法中添加以下内容:

dispatch_release(taskQ);

dispatch_release(taskQ);

And add this as an ivar in your class declaration:

并将其作为ivar添加到你的类声明中:

dispatch_queue_t taskQ;

dispatch_queue_t taskQ;

#2

I believe Ryan is on the right path: there are simply too many threads being spawned when a project has 1,500 files (the amount I decided to test with.)

我相信Ryan的方法是正确的:当一个项目有1500个文件时(我决定用这个数量进行测试)，就会产生太多的线程。

So, I refactored the code above to work like this:

因此，我重构了上面的代码如下所示:

- (void) establishImportLinksForFilesInProject:(LPProject *)aProject
{
        dispatch_queue_t taskQ = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);

     dispatch_async(taskQ, 
     ^{

     // Create a new Core Data Context on this thread using the same persistent data store    
     // as the main thread. Pass the objectID of aProject to access the managedObject
     // for that project on this thread's context:

     NSManagedObjectID *projectID = [aProject objectID];

     for (LPFile *fileToCheck in [backgroundContext objectWithID:projectID] memberFiles])
     {
        if (//Some condition is met)
        {
                // Here, we do the scanning for @import statements. 
                // When we find a valid one, we put the whole path to the 
                // imported file into an array called 'verifiedImports'. 

                // Pass this ID to main thread in dispatch call below to access the same
                // file in the main thread's context
                NSManagedObjectID *fileID = [fileToCheck objectID];


                // go back to the main thread and update the model 
                // (Core Data is not thread-safe.)
                dispatch_async(dispatch_get_main_queue(), 
                ^{
                    for (NSString *import in verifiedImports)
                    {  
                       LPFile *targetFile = [mainContext objectWithID:fileID];
                       // Add the relationship to targetFile. 
                    }
                 });//end block
         }
    }
    // Easy way to tell when we're done processing all files.
    // Could add a dispatch_async(main_queue) call here to do something like UI updates, etc

    });//end block
    }

So, basically, we're now spawning one thread that reads all the files instead of one-thread-per-file. Also, it turns out that calling dispatch_async() on the main_queue is the correct approach: the worker thread will dispatch that block to the main thread and NOT wait for it to return before proceeding to scan the next file.

因此，基本上，我们现在生成了一个线程，它读取所有文件，而不是每个文件一个线程。此外，在main_queue上调用dispatch_async()也是正确的方法:工作线程将把这个块分派给主线程，而不是等待它返回，然后再继续扫描下一个文件。

This implementation essentially sets up a "serial" queue as Ryan suggested (the for loop is the serial part of it), but with one advantage: when the for loop ends, we're done processing all the files and we can just stick a dispatch_async(main_queue) block there to do whatever we want. It's a very nice way to tell when the concurrent processing task is finished and that didn't exist in my old version.

这个实现实际上按照Ryan的建议设置了一个“串行”队列(for循环是它的串行部分)，但是有一个优点:当for循环结束时，我们完成了对所有文件的处理，并且我们可以在其中插入一个dispatch_async(main_queue)块，以便做我们想做的任何事情。这是一种很好的方式来判断何时完成并发处理任务，而在我的旧版本中并不存在。

The disadvantage here is that it's a bit more complicated to work with Core Data on multiple threads. But this approach seems to be bulletproof for projects with 5,000 files (which is the highest I've tested.)

这里的缺点是在多个线程上处理核心数据有点复杂。但是这种方法对于有5000个文件的项目来说似乎是防弹的(这是我测试过的最高的)。

#3

I think it is more easy to understand with diagram:

我认为用图表更容易理解:

For the situation the author described:

对于作者所描述的情况:

|taskQ| ***********start|

| taskQ | * * * * * * * * * * *开始|

|dispatch_1 ***********|---------

| dispatch_1 * * * * * * * * * * * | - - - - - - - - - - - -

|dispatch_2 *************|---------

| dispatch_2 * * * * * * * * * * * * * | - - - - - - - - - - - -

.

。

|dispatch_n ***************************|----------

| dispatch_n * * * * * * * * * * * * * * * * * * * * * * * * * * * | - - - - - - - - - - -

|main queue(sync)|**start to dispatch to main|

|主队列(sync)|*开始分派到主|

*************************|--dispatch_1--|--dispatch_2--|--dispatch3--|*****************************|--dispatch_n|,

* * * * * * * * * * * * * * * * * * * * * * * * * |——dispatch_1 |——dispatch_2 |——dispatch3 | * * * * * * * * * * * * * * * * * * * * * * * * * * * * * |——dispatch_n |,

which make the sync main queue so busy that finally fail the task.

这使得同步主队列非常繁忙，最终导致任务失败。