如何强制Dask Worker不要将数据写入磁盘?

时间:2019-03-15 16:55:55

标签: dask dask-distributed

我试图强迫一个工作人员不要将数据写入磁盘,但是可能是我做错了什么。

我对文件〜/ .config / dask / distributed.yaml进行了如下修改:

<div class="shopping-cart">
   <header class="shopping-cart__header">
     <span class="shopping-cart__header__label-price">Price</span>
     <span class="shopping-cart__header__label-quantity">Quantity</span>
     <span class="shopping-cart__header__label-total">Total</span>
   </header>
    <div class="product-row">

        <div class="product-image">
                <img src="https://www.drivencoffee.com/wp-content/uploads/2016/03/Scandinavian-Blend-coffee.jpg" />
        </div>

        <div class="product-details">
            <div class="product-title">Coffee</div>
            <div class="product-description">Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim  </div>
        </div>
        <div class="product-price">12.99</div>
        <div class="product-quantity">
            <input type="number" value="2" min="1" />
        </div>

        <div clas="product-removal">
            <button class="remove-product">
                Remove
            </button>
        </div>

        <div class="product-total-price">
            25.98
        </div>
    </div>




</div>

但是他们只是继续将任何临时结果写入磁盘。我也尝试过使用虚拟内存限制设置(100GB),但是在任务结束时它们仍然写在磁盘上。我该如何强迫他们保留所有记忆? 我想念什么吗?

1 个答案:

答案 0 :(得分:1)

您在做什么应该可以正常工作。我很惊讶它不适合您。

官方建议使用false值,如此处所述:https://docs.dask.org/en/latest/setup/hpc.html#no-local-storage

distributed:
  worker:
    memory:
      target: false  # don't spill to disk
      spill: false  # don't spill to disk
      pause: 0.80  # pause execution at 80% memory use
      terminate: 0.95  # restart the worker at 95% use