编程中需要用到 Python(2.7) 的 paramiko 模块下载服务器上面一个比较大的文件,命令如下:
client = paramiko.Transport(ip, port)
client.connect(username=user, password=password)
sftp = paramiko.SFTPClient.from_transport(client)
sftp.get(remotefile, localfile)
发现一个很奇怪的现象,下载比较小的文件时,没有发现异常,直到有一次下载一个 300mb+的文件时,中途卡住一直下载不下来
解决方案:
修改 python 有关 sftp 数据下载的源码,文件位置为:/usr/lib/python2.7/site-packages/paramiko/sftp_file.py
找到_prefetch_thread 函数:
def _prefetch_thread(self, chunks):
# do these read requests in a temporary thread because there may be
# a lot of them, so it may block.
for offset, length in chunks:
num = self.sftp._async_request(
self,
CMD_READ,
self.handle,
long(offset),
int(length))
with self._prefetch_lock:
self._prefetch_extents[num] = (offset, length)
代码作者注释说明了此处可能存在并发过大的问题,所以我们对他进行修改:
def _prefetch_thread(self, chunks):
# do these read requests in a temporary thread because there may be
# a lot of them, so it may block.
max_request_num = 512
to_wait = False
for offset, length in chunks:
num = self.sftp._async_request(
self, CMD_READ, self.handle, long(offset), int(length)
)
with self._prefetch_lock:
self._prefetch_extents[num] = (offset, length)
if len(self._prefetch_extents) >= max_request_num:
to_wait = True
if to_wait:
time.sleep(1)
to_wait = False
总体思路就是限制同时存在的异步请求总数
博主真是太厉害了!!!
真好呢
博主太厉害了!