我想使用spring-webClient下载文件（600MB），并且我同时下载六个文件。然后，应用程序获得一个异常（io.netty.util.internal.OutOfDirectMemoryError），我猜它在下载文件时所有数据都在内存中。但是我不知道如何用spring-webclient解决它。
当我同时下载一个文件时，这没问题，但是下载多个文件时会出现异常。
public static Mono<Resource> getUrlResource(String url, int depth) {
if (depth >= 10) {
return Mono.empty();
}
return WebClient.builder().build().get().uri(url)
.exchange()
.flatMap(clientResponse -> {
if (clientResponse.statusCode().is3xxRedirection()) {
String redirectUrl = clientResponse.headers().header("Location").get(0);
return clientResponse.bodyToMono(Void.class).then(getUrlResource(redirectUrl, depth + 1));
}
return clientResponse.bodyToMono(Resource.class);
});
}
public static File getFileFromOssUrl(String url, String destFilePath) {
Mono<Resource> fileMono = getUrlResource(url, 1);
Resource resource = fileMono.block();
File tempFile = null;
if (resource != null) {
try {
tempFile = new File(destFilePath);
if (tempFile.exists()) {
tempFile.delete();
}
tempFile.createNewFile();
FileCopyUtils.copy(resource.getInputStream(), new BufferedOutputStream(new FileOutputStream(tempFile)));
} catch (IOException e) {
e.printStackTrace();
}
}
return tempFile;
}
该行中发生异常
Resource resource = fileMono.block();
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3808428039, max: 3817865216)
reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3808428039, max: 3817865216)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.FluxMap] :
reactor.core.publisher.Flux.map(Flux.java:5732)
reactor.netty.ByteBufFlux.fromInbound(ByteBufFlux.java:71)
Error has been observed by the following operator(s):
|_ Flux.map ⇢ reactor.netty.ByteBufFlux.fromInbound(ByteBufFlux.java:71)
|_ Flux.doOnSubscribe ⇢ org.springframework.http.client.reactive.ReactorClientHttpResponse.getBody(ReactorClientHttpResponse.java:64)
|_ Flux.doOnCancel ⇢ org.springframework.http.client.reactive.ReactorClientHttpResponse.getBody(ReactorClientHttpResponse.java:69)
|_ Flux.map ⇢ org.springframework.http.client.reactive.ReactorClientHttpResponse.getBody(ReactorClientHttpResponse.java:75)
|_ Flux.collectList ⇢ org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:445)
|_ Mono.filter ⇢ org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:446)
|_ Mono.map ⇢ org.springframework.core.io.buffer.DataBufferUtils.join(DataBufferUtils.java:447)
|_ Mono.map ⇢ org.springframework.core.codec.AbstractDataBufferDecoder.decodeToMono(AbstractDataBufferDecoder.java:68)
|_ Mono.flatMap ⇢ com.gaosiedu.mw.videowonderfulediting.base.util.StorageUtil.getUrlResource(StorageUtil.java:23)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:93)
at reactor.core.publisher.Mono.block(Mono.java:1494)
at com.gaosiedu.mw.videowonderfulediting.base.util.StorageUtil.getFileFromOssUrl(StorageUtil.java:34)
at com.gaosiedu.mw.videowonderfulediting.business.JobXunfeiService.addTask(JobXunfeiService.java:93)
at com.gaosiedu.mw.videowonderfulediting.business.JobXunfeiService$$FastClassBySpringCGLIB$$3a9ba562.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.gaosiedu.mw.videowonderfulediting.business.JobXunfeiService$$EnhancerBySpringCGLIB$$e6579dec.addTask(<generated>)
at com.gaosiedu.mw.videowonderfulediting.xunfei.schedule.NewJobSubmitSchedule.addNewJob(NewJobSubmitSchedule.java:39)
at com.gaosiedu.mw.videowonderfulediting.xunfei.schedule.NewJobSubmitSchedule$$FastClassBySpringCGLIB$$1b1c01d9.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 3808428039, max: 3817865216)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:667)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:622)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:772)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:748)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:215)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:147)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)