如何交错/合并异步迭代？

时间：2018-05-29 13:19:22

标签： javascript promise iterator async-await

假设我有一些像这样的asnyc可迭代对象：

// Promisified sleep function
const sleep = ms => new Promise((resolve, reject) => {
  setTimeout(() => resolve(ms), ms);
});

const a = {
  [Symbol.asyncIterator]: async function * () {
    yield 'a';
    await sleep(1000);
    yield 'b';
    await sleep(2000);
    yield 'c';
  }, 
};

const b = {
  [Symbol.asyncIterator]: async function * () {
    await sleep(6000);
    yield 'i';
    yield 'j';
    await sleep(2000);
    yield 'k';
  }, 
};

const c = {
  [Symbol.asyncIterator]: async function * () {
    yield 'x';
    await sleep(2000);
    yield 'y';
    await sleep(8000);
    yield 'z';
    await sleep(10000);
    throw new Error('You have gone too far! ');
  }, 
};

现在，假设我可以像这样结束他们：

const abcs = async function * () {
  yield * a;
  yield * b;
  yield * c;
};

产生的（前9个）项目将是：

(async () => {
  const limit = 9;
  let i = 0; 
  const xs = [];
  for await (const x of abcs()) {
    xs.push(x);
    i++;
    if (i === limit) {
      break;
    }
  }
  console.log(xs);
})().catch(error => console.error(error));

// [ 'a', 'b', 'c', 'i', 'j', 'k', 'x', 'y', 'z' ]

但是想象一下，我不关心订单，a，b和c以不同的速度屈服，我想要< em>尽快收益。

如何重写此循环以尽快产生x，忽略顺序？

a，b或c也可能是无限序列，因此解决方案不能要求将所有元素缓冲到数组中。

6 个答案:

答案 0 :(得分：4)

没有办法用循环语句写这个。 async / await代码总是按顺序执行，以便同时执行需要直接使用promise combinators的事情。对于简单的承诺，有Promise.all，对于异步迭代器，还没有（还），所以我们需要自己编写它：

async function* combine(iterable) {
    const asyncIterators = Array.from(iterable, o => o[Symbol.asyncIterator]());
    const results = [];
    let count = asyncIterators.length;
    const never = new Promise(() => {});
    function getNext(asyncIterator, index) {
        return asyncIterator.next().then(result => ({
            index,
            result,
        }));
    }
    const nextPromises = asyncIterators.map(getNext);
    while (count) {
        const {index, result} = await Promise.race(nextPromises);
        if (result.done) {
            nextPromises[index] = never;
            results[index] = result.value;
            count--;
        } else {
            nextPromises[index] = getNext(asyncIterators[index], index);
            yield result.value;
        }
    }
    return results;
}

请注意，combine不支持将值传递到next或通过.throw或.return取消。

您可以将其称为

(async () => {
  for await (const x of combine([a, b, c])) {
    console.log(x);
  }
})().catch(console.error);

答案 1 :(得分：2)

如果我更改abcs以接受要处理的生成器，我会想到这一点，请参阅内联注释：

const abcs = async function * (...gens) {
  // Worker function to queue up the next result
  const queueNext = async (e) => {
    e.result = null; // Release previous one as soon as possible
    e.result = await e.it.next();
    return e;
  };
  // Map the generators to source objects in a map, get and start their
  // first iteration
  const sources = new Map(gens.map(gen => [
    gen,
    queueNext({
      key: gen,
      it:  gen[Symbol.asyncIterator]()
    })
  ]));
  // While we still have any sources, race the current promise of
  // the sources we have left
  while (sources.size) {
    const winner = await Promise.race(sources.values());
    // Completed the sequence?
    if (winner.result.done) {
      // Yes, drop it from sources
      sources.delete(winner.key);
    } else {
      // No, grab the value to yield and queue up the next
      // Then yield the value
      const {value} = winner.result;
      sources.set(winner.key, queueNext(winner));
      yield value;
    }
  }
};

直播示例：

＆＃13;
＆＃13; 
// Promisified sleep function
const sleep = ms => new Promise((resolve, reject) => {
  setTimeout(() => resolve(ms), ms);
});

const a = {
  [Symbol.asyncIterator]: async function * () {
    yield 'a';
    await sleep(1000);
    yield 'b';
    await sleep(2000);
    yield 'c';
  }, 
};

const b = {
  [Symbol.asyncIterator]: async function * () {
    await sleep(6000);
    yield 'i';
    yield 'j';
    await sleep(2000);
    yield 'k';
  }, 
};

const c = {
  [Symbol.asyncIterator]: async function * () {
    yield 'x';
    await sleep(2000);
    yield 'y';
    await sleep(8000);
    yield 'z';
  }, 
};

const abcs = async function * (...gens) {
  // Worker function to queue up the next result
  const queueNext = async (e) => {
    e.result = null; // Release previous one as soon as possible
    e.result = await e.it.next();
    return e;
  };
  // Map the generators to source objects in a map, get and start their
  // first iteration
  const sources = new Map(gens.map(gen => [
    gen,
    queueNext({
      key: gen,
      it:  gen[Symbol.asyncIterator]()
    })
  ]));
  // While we still have any sources, race the current promise of
  // the sources we have left
  while (sources.size) {
    const winner = await Promise.race(sources.values());
    // Completed the sequence?
    if (winner.result.done) {
      // Yes, drop it from sources
      sources.delete(winner.key);
    } else {
      // No, grab the value to yield and queue up the next
      // Then yield the value
      const {value} = winner.result;
      sources.set(winner.key, queueNext(winner));
      yield value;
    }
  }
};

(async () => {
  console.log("start");
  for await (const x of abcs(a, b, c)) {
    console.log(x);
  }
  console.log("done");
})().catch(error => console.error(error));
＆＃13; 
.as-console-wrapper {
  max-height: 100% !important;
}
＆＃13;
＆＃13;
＆＃13;

答案 2 :(得分：1)

如果有人觉得它有用，这里是当前 accepted answer 的打字稿版本：


const combineAsyncIterables = async function* <T>(
  asyncIterables: AsyncIterable<T>[],
): AsyncGenerator<T> {
  const asyncIterators = Array.from(asyncIterables, (o) =>
    o[Symbol.asyncIterator](),
  );
  const results = [];
  let count = asyncIterators.length;
  const never: Promise<never> = new Promise(noOp);
  const getNext = (asyncIterator: AsyncIterator<T>, index: number) =>
    asyncIterator.next().then((result) => ({ index, result }));

  const nextPromises = asyncIterators.map(getNext);
  try {
    while (count) {
      const { index, result } = await Promise.race(nextPromises);
      if (result.done) {
        nextPromises[index] = never;
        results[index] = result.value;
        count--;
      } else {
        nextPromises[index] = getNext(asyncIterators[index], index);
        yield result.value;
      }
    }
  } finally {
    for (const [index, iterator] of asyncIterators.entries()) {
      if (nextPromises[index] != never && iterator.return != null) {
        // no await here - see https://github.com/tc39/proposal-async-iteration/issues/126
        void iterator.return();
      }
    }
  }
  return results;
};

答案 3 :(得分：0)

我使用异步生成器解决了这个问题。 （我希望我几天前能找到这个问题，这会节省我一些时间） 很乐意听取意见和批评。

async function* mergen(...gens) {
  const promises = gens.map((gen, index) =>
    gen.next().then(p => ({...p, gen}))
  );

  while (promises.length > 0) {
    yield race(promises).then(({index, value: {value, done, gen}}) => {
      promises.splice(index, 1);
      if (!done)
        promises.push(
          gen.next().then(({value: newVal, done: newDone}) => ({
            value: newVal,
            done: newDone,
            gen
          }))
        );
      return value;
    });
  }
};

// Needed to implement race to provide index of resolved promise
function race(promises) {
  return new Promise(resolve =>
    promises.forEach((p, index) => {
      p.then(value => {
        resolve({index, value});
      });
    })
  );
}

我花了很多时间才找到，我很兴奋，我把它放在一个npm包中:) https://www.npmjs.com/package/mergen

答案 4 :(得分：0)

这是一项复杂的任务，所以我将其分解为各个部分：

步骤1：将每个可迭代的值记录到控制台中

在考虑创建异步迭代器之前，我们首先应该考虑将每个迭代器到达时将每个值简单地记录到控制台的任务。与javascript中的大多数并发任务一样，这涉及调用多个异步函数并使用Promise.all等待它们的结果。

function merge(iterables) {
  return Promise.all(
    Array.from(iterables).map(async (iter) => {
      for await (const value of iter) {
        console.log(value);
      }
    }),
  );
}

// a, b and c are the async iterables defined in the question
merge([a, b, c]); // a, x, b, y, c, i, j, k, z, Error: you have gone too far!

CodeSandbox链接：https://codesandbox.io/s/tender-ives-4hijy?fontsize=14

merge函数记录来自每个迭代器的值，但是几乎没有用；当所有迭代器完成时，它将返回一个承诺，该承诺应满足undefined数组。

第2步：用合并异步生成器替换合并功能

下一步是将console.log调用替换为对推入父级异步迭代器的函数的调用。要使用异步生成器执行此操作，我们需要更多代码，因为将值“推送”到异步生成器的唯一方法是使用yield运算符，该运算符不能在子函数作用域中使用。解决方案是创建两个队列，一个推入队列和一个拉入队列。接下来，我们定义一个push函数，该函数将在没有挂起的拉取时推送到推入队列，或者将一个值排队以待稍后拉取。最终，我们必须永久地从推队列中产生值（如果它具有值），或者保证将一个解析函数加入队列，以便稍后推入调用。这是代码：

async function *merge(iterables) {
  // pushQueue and pullQueue will never both contain values at the same time.
  const pushQueue = [];
  const pullQueue = [];
  function push(value) {
    if (pullQueue.length) {
      pullQueue.pop()(value);
    } else {
      pushQueue.unshift(value);
    }
  }

  // the merge code from step 1
  const finishP = Promise.all(
    Array.from(iterables).map(async (iter) => {
      for await (const value of iter) {
        push(value);
      }
    }),
  );

  while (true) {
    if (pushQueue.length) {
      yield pushQueue.pop();
    } else {
      // important to note that yield in an async generator implicitly awaits promises.
      yield new Promise((resolve) => {
        pullQueue.unshift(resolve);
      });
    }
  }
}

// code from the question
(async () => {
  const limit = 9;
  let i = 0; 
  const xs = [];
  for await (const x of merge([a, b, c])) {
    xs.push(x);
    console.log(x);
    i++;
    if (i === limit) {
      break;
    }
  }
  console.log(xs); // ["a", "x", "b", "y", "c", "i", "j", "k", "z"]
})().catch(error => console.error(error));

CodeSandbox链接：https://codesandbox.io/s/misty-cookies-du1eg

这几乎可行！如果运行代码，您会注意到xs的打印正确，但是break语句未得到遵守，并且值继续从子迭代器中提取，从而导致在{{ 1}}被抛出，导致未处理的承诺被拒绝。另请注意，我们对c调用的结果不做任何事情。理想情况下，当Promise.all承诺成立时，应返回生成器。我们只需要一点点代码来确保1.返回父迭代器时返回子迭代器（例如，在finishP循环中使用break语句），并2。所有子迭代器都返回时，返回父迭代器。

步骤3：在返回父迭代器时停止每个子迭代器，并在每个孩子返回后停止父迭代器。

为确保在返回父异步生成器时正确返回每个子异步iterable，我们可以使用finally块侦听父异步生成器的完成。为了确保在子迭代器返回时返回父生成器，我们可以将产生的承诺与for await承诺进行竞争。

finishP

CodeSandbox链接：https://codesandbox.io/s/vigilant-leavitt-h247u

在此代码投入生产之前，我们仍然需要做一些事情。例如，从子迭代器连续提取值，而无需等待父迭代器提取它们。结合async function *merge(iterables) { const pushQueue = []; const pullQueue = []; function push(value) { if (pullQueue.length) { pullQueue.pop()(value); } else { pushQueue.unshift(value); } } // we create a promise to race calls to iter.next let stop; const stopP = new Promise((resolve) => (stop = resolve)); let finished = false; const finishP = Promise.all( Array.from(iterables).map(async (iter) => { // we use the iterator interface rather than the iterable interface iter = iter[Symbol.asyncIterator](); try { while (true) { // because we can’t race promises with for await, we have to call iter.next manually const result = await Promise.race([stopP, iter.next()]); if (!result || result.done) { return; } push(result.value); } } finally { // we should be a good citizen and return child iterators await iter.return && iter.return(); } }), ).finally(() => (finished = true)); try { while (!finished) { if (pushQueue.length) { yield pushQueue.pop(); } else { const value = await Promise.race([ new Promise((resolve) => { pullQueue.unshift(resolve); }), finishP, ]); if (!finished) { yield value; } } } // we await finishP to make the iterator catch any promise rejections await finishP; } finally { stop(); } } 是一个无界数组的事实，如果父迭代器以比子迭代器生成它们更慢的速度拉取值，则可能导致内存泄漏。

此外，合并迭代器返回pushQueue作为其最终值，但是您可能希望最终值是最后完成的子迭代器中的最终值。

如果您正在寻找一个小型的，集中的库，该库具有类似于上面的合并功能，其中涵盖了更多用例和边缘用例，请查看我写的Repeater.js。它定义了静态方法Repeater.merge，该方法如上所述。它还提供了一个干净的API，用于将基于回调的API转换为Promise和其他组合器静态方法，以其他方式组合异步迭代器。

答案 5 :(得分：-1)

我希望我能正确理解你的问题，这就是我如何处理它：

let results = [];

Promise.all([ a, b, c ].map(async function(source) {
    for await (let item of source) {
        results.push(item);
    }
}))
.then(() => console.log(results));

我尝试了三个普通数组：

var a = [ 1, 2, 3 ];
var b = [ 4, 5, 6 ];
var c = [ 7, 8, 9 ];

结果是[1, 4, 7, 2, 5, 8, 3, 6, 9]。

相关问题
最新问题