Scala / Akka：为什么期货期货在Akka Dispatcher

时间：2018-04-06 14:15:09

标签： scala parallel-processing akka future akka-dispatcher

当我们尝试从actor的receive方法中启动一些期货时，我们发现了一种奇怪的行为。 如果我们使用配置的调度程序作为ExecutionContext，则期货在同一个线程上按顺序运行。如果我们使用ExecutionContext.Implicits.global，期货将按预期并行运行。

我们将代码简化为以下示例（下面是一个更完整的示例）：

implicit val ec = context.getDispatcher

Future{ doWork() } // <-- all running parallel
Future{ doWork() }
Future{ doWork() }
Future{ doWork() }

Future {
   Future{ doWork() } 
   Future{ doWork() } // <-- NOT RUNNING PARALLEL!!! WHY!!!
   Future{ doWork() }
   Future{ doWork() }
}

可编辑的例子是这样的：

import akka.actor.ActorSystem
import scala.concurrent.{ExecutionContext, Future}

object WhyNotParallelExperiment extends App {

  val actorSystem = ActorSystem(s"Experimental")   

  // Futures not started in future: running in parallel
  startFutures(runInFuture = false)(actorSystem.dispatcher)
  Thread.sleep(5000)

  // Futures started in future: running in sequentially. Why????
  startFutures(runInFuture = true)(actorSystem.dispatcher)
  Thread.sleep(5000)

  actorSystem.terminate()

  private def startFutures(runInFuture: Boolean)(implicit executionContext: ExecutionContext): Unit = {
    if (runInFuture) {
      Future{
        println(s"Start Futures on thread ${Thread.currentThread().getName()}")
        (1 to 9).foreach(startFuture)
        println(s"Started Futures on thread ${Thread.currentThread().getName()}")
      }
    } else {
      (11 to 19).foreach(startFuture)
    }
  }

  private def startFuture(id: Int)(implicit executionContext: ExecutionContext): Future[Unit] = Future{
    println(s"Future $id should run for 500 millis on thread ${Thread.currentThread().getName()}")
    Thread.sleep(500)
    println(s"Future $id finished on thread ${Thread.currentThread().getName()}")
  }


}

我们尝试使用thread-pool-executor和fork-join-executor，结果相同。

我们是否以错误的方式使用期货？ 那么你应该如何产生并行任务呢？

3 个答案:

答案 0 :(得分：2)

来自Akka的内部BatchingExecutor（强调我的）的描述：

  

执行者的混合特征，将多个嵌套Runnable.run()调用分组到一个传递给原始执行者的Runnable 。这可能是一个有用的优化，因为它绕过原始上下文的任务队列并将相关（嵌套）代码保存在单个线程上，这可以提高CPU亲和性。但是，如果传递给Executor的任务是阻塞或昂贵的，那么这种优化可以防止工作窃取并使性能变得更糟......如果代码不应该使用scala.concurrent.blocking，那么批处理执行程序可以创建死锁，因为任务在其他任务中创建将阻止外部任务完成。

如果您正在使用混合在BatchingExecutor中的调度程序 - 即MessageDispatcher的子类 - 您可以使用scala.concurrent.blocking构造来启用与嵌套Futures的并行性：< / p> 

Future {
  Future {
    blocking {
      doBlockingWork()
    }
  }
}

在您的示例中，您需要在blocking方法中添加startFuture：

private def startFuture(id: Int)(implicit executionContext: ExecutionContext): Future[Unit] = Future {
  blocking {
    println(s"Future $id should run for 500 millis on thread ${Thread.currentThread().getName()}")
    Thread.sleep(500)
    println(s"Future $id finished on thread ${Thread.currentThread().getName()}")
  }
}

通过以上更改运行startFutures(true)(actorSystem.dispatcher)的示例输出：

Start Futures on thread Experimental-akka.actor.default-dispatcher-2
Started Futures on thread Experimental-akka.actor.default-dispatcher-2
Future 1 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-2
Future 3 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-3
Future 5 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-6
Future 7 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-7
Future 4 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-5
Future 9 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-10
Future 6 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-8
Future 8 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-9
Future 2 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-4
Future 1 finished on thread Experimental-akka.actor.default-dispatcher-2
Future 3 finished on thread Experimental-akka.actor.default-dispatcher-3
Future 5 finished on thread Experimental-akka.actor.default-dispatcher-6
Future 4 finished on thread Experimental-akka.actor.default-dispatcher-5
Future 8 finished on thread Experimental-akka.actor.default-dispatcher-9
Future 7 finished on thread Experimental-akka.actor.default-dispatcher-7
Future 9 finished on thread Experimental-akka.actor.default-dispatcher-10
Future 6 finished on thread Experimental-akka.actor.default-dispatcher-8
Future 2 finished on thread Experimental-akka.actor.default-dispatcher-4

答案 1 :(得分：1)

它与调度员的“吞吐量”设置有关。我在application.conf中添加了一个“公平调度程序”来演示：

fair-dispatcher {
  # Dispatcher is the name of the event-based dispatcher
  type = Dispatcher
  # What kind of ExecutionService to use
  executor = "fork-join-executor"
  # Configuration for the fork join pool
  fork-join-executor {
    # Min number of threads to cap factor-based parallelism number to
    parallelism-min = 2
    # Parallelism (threads) ... ceil(available processors * factor)
    parallelism-factor = 2.0
    # Max number of threads to cap factor-based parallelism number to
    parallelism-max = 10
  }
  # Throughput defines the maximum number of messages to be
  # processed per actor before the thread jumps to the next actor.
  # Set to 1 for as fair as possible.
  throughput = 1
}

以下是您对Futures使用公平调度程序进行一些修改并打印吞吐量设置当前值的示例：

package com.test

import akka.actor.ActorSystem

import scala.concurrent.{ExecutionContext, Future}

object WhyNotParallelExperiment extends App {

  val actorSystem = ActorSystem(s"Experimental")

  println("Default dispatcher throughput:")
  println(actorSystem.dispatchers.defaultDispatcherConfig.getInt("throughput"))

  println("Fair dispatcher throughput:")
  println(actorSystem.dispatchers.lookup("fair-dispatcher").configurator.config.getInt("throughput"))

  // Futures not started in future: running in parallel
  startFutures(runInFuture = false)(actorSystem.dispatcher)
  Thread.sleep(5000)

  // Futures started in future: running in sequentially. Why????
  startFutures(runInFuture = true)(actorSystem.dispatcher)
  Thread.sleep(5000)

  actorSystem.terminate()

  private def startFutures(runInFuture: Boolean)(implicit executionContext: ExecutionContext): Unit = {
    if (runInFuture) {
      Future{
        implicit val fairExecutionContext = actorSystem.dispatchers.lookup("fair-dispatcher")
        println(s"Start Futures on thread ${Thread.currentThread().getName()}")
        (1 to 9).foreach(i => startFuture(i)(fairExecutionContext))
        println(s"Started Futures on thread ${Thread.currentThread().getName()}")
      }
    } else {
      (11 to 19).foreach(startFuture)
    }
  }

  private def startFuture(id: Int)(implicit executionContext: ExecutionContext): Future[Unit] = Future{
    println(s"Future $id should run for 500 millis on thread ${Thread.currentThread().getName()}")
    Thread.sleep(500)
    println(s"Future $id finished on thread ${Thread.currentThread().getName()}")
  }
}

输出：

Default dispatcher throughput:
5
Fair dispatcher throughput:
1
Future 12 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-3
Future 11 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-4
Future 13 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-2
Future 14 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-5
Future 16 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-7
Future 15 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-6
Future 17 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-8
Future 18 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-9
Future 19 should run for 500 millis on thread Experimental-akka.actor.default-dispatcher-10
Future 13 finished on thread Experimental-akka.actor.default-dispatcher-2
Future 11 finished on thread Experimental-akka.actor.default-dispatcher-4
Future 12 finished on thread Experimental-akka.actor.default-dispatcher-3
Future 14 finished on thread Experimental-akka.actor.default-dispatcher-5
Future 16 finished on thread Experimental-akka.actor.default-dispatcher-7
Future 15 finished on thread Experimental-akka.actor.default-dispatcher-6
Future 17 finished on thread Experimental-akka.actor.default-dispatcher-8
Future 18 finished on thread Experimental-akka.actor.default-dispatcher-9
Future 19 finished on thread Experimental-akka.actor.default-dispatcher-10
Start Futures on thread Experimental-akka.actor.default-dispatcher-10
Future 1 should run for 500 millis on thread Experimental-fair-dispatcher-12
Future 2 should run for 500 millis on thread Experimental-fair-dispatcher-13
Future 4 should run for 500 millis on thread Experimental-fair-dispatcher-15
Future 3 should run for 500 millis on thread Experimental-fair-dispatcher-14
Future 5 should run for 500 millis on thread Experimental-fair-dispatcher-17
Future 6 should run for 500 millis on thread Experimental-fair-dispatcher-16
Future 7 should run for 500 millis on thread Experimental-fair-dispatcher-18
Future 8 should run for 500 millis on thread Experimental-fair-dispatcher-19
Started Futures on thread Experimental-akka.actor.default-dispatcher-10
Future 4 finished on thread Experimental-fair-dispatcher-15
Future 2 finished on thread Experimental-fair-dispatcher-13
Future 1 finished on thread Experimental-fair-dispatcher-12
Future 9 should run for 500 millis on thread Experimental-fair-dispatcher-15
Future 5 finished on thread Experimental-fair-dispatcher-17
Future 7 finished on thread Experimental-fair-dispatcher-18
Future 8 finished on thread Experimental-fair-dispatcher-19
Future 6 finished on thread Experimental-fair-dispatcher-16
Future 3 finished on thread Experimental-fair-dispatcher-14
Future 9 finished on thread Experimental-fair-dispatcher-15

正如您所看到的，公平调度员在大多数期货中使用不同的线程。

默认调度程序针对actor进行了优化，因此将吞吐量设置为5以最小化上下文切换以提高消息处理吞吐量，同时保持一定程度的公平性。

我的公平调度程序中唯一的变化是吞吐量：1，即如果可能，每个异步执行请求都有自己的线程（最多为parallelism-max）。

我建议为不同用途的期货创建单独的调度程序。例如。一个调度程序（即线程池）用于调用某些Web服务，另一个用于阻止数据库访问等。这可以通过调整自定义调度程序设置来更精确地控制它。

看一下https://doc.akka.io/docs/akka/current/dispatchers.html，它对于理解细节非常有用。

同时查看Akka参考设置（特别是默认调度员），那里有很多有用的评论：https://github.com/akka/akka/blob/master/akka-actor/src/main/resources/reference.conf

答案 2 :(得分：0)

经过一番研究后，我发现Dispatcher类实现akka.dispatch.BatchingExecutor。出于性能原因，此类检查应在同一线程上批处理哪些任务。 Future.map在内部创建scala.concurrent.OnCompleteRunnable，该BatchingExecutor在map()。

中进行批处理

flatMap() / Future.apply这似乎是合理的，其中一个任务生成一个后续任务，但不适用于用于分叉工作的显式新Futures。 在内部，Future.successful().map由object MyFuture { def apply[T](body: =>T)(implicit executor: ExecutionContext): Future[T] = { val promise = Promise[T]() class FuturesStarter extends Runnable { override def run(): Unit = { promise.complete(Try(body)) } } executor.execute(new FuturesStarter) promise.future } } 实现，因此被批量处理。我的解决方法是以不同的方式创建未来：

FutureStarter

Future - Runnables没有批处理，因此并行运行。

有人可以确认这个解决方案没问题吗？ 有没有更好的方法来解决这个问题？ 需要BatchingExecutor / moduleForComponent的当前实现，还是错误？

相关问题
最新问题