[{"data":1,"prerenderedAt":230},["ShallowReactive",2],{"article-how-to-download-big-files":3},{"title":4,"article":5,"hot_article":16,"month_arch":79,"cover":227},"大文件下载的一些经验 - 独孤影的博客",{"id":6,"title":7,"uri":8,"author":9,"tags":10,"created_at":13,"view_count":14,"content":15},69,"大文件下载的一些经验","how-to-download-big-files","duguying",[11,12],"下载","cdn","2022-08-26 15:32:21",488,"\u003Cp>数年之前，我在一家CDN公司上班。在CDN项目主要工作是负责对客户视频数据搬运到云存储，中间也涉及到下载转码上传之类的工作。其中踩过一些坑，也吸取了一些经验。下面简单讲一下我遇到的几个问题。\u003C\u002Fp>\n\n\u003Ch3>下载没网速了\u003C\u002Fh3>\n\n\u003Cp>首先是下载没网速的情况。在处理大量的下载任务过程中，发现了一些特殊情况，客户通知我的服务回源，我的下载服务从源站下载一些大文件的时候，开始网络一切正常，可是逐渐的网速越来越低，甚至没网速，这回导致下载一直卡主。可能你会说，设置一个下载超时不就好了，但是，由于客户的文件大多是影视原片，体积巨大，有些甚至达到了几十G，如果简单的设置超时，以处理时间判断下载成败，那么这些大文件基本上不可能成功，都会以超时失败。当然，也可以分片下载来解决文件大的问题，但是对于0网速或者只有几B\u002Fs的网速，你的分片就显得无力了。其实，这个状况产生的原因大多是因为源站故障导致的，比如负载高之类的。只要断开连接重新下载就有机会恢复正常速度的连接，而且一般源站是有多个的，重新连接到新的节点就好了。所以，这里“断开连接重新发起下载”这个能力就非常重要了。我原本使用的是\u003Ccode class=\"inline-code\">github.com\u002Fnareix\u002Fcurl\u003C\u002Fcode>这个库进行文件下载，它的优点是创建异步下载任务后，可以监控当前下载的速度。这样，我可以对每个下载任务的网速进行监控，当网速低于一个阈值的时候，就放弃下载并重试。最初的时候只是简单的调用它的\u003Ccode class=\"inline-code\">Close()\u003C\u002Fcode>方法，关闭了下载。后面却发现，经常发生fd用完的情况，一查才知道\u003Ccode class=\"inline-code\">Close()\u003C\u002Fcode>方法并不能释放连接。最后只好自己修改了这个库，给它添加了强制关闭连接的能力。于是就完美的解决了这一问题。记得当时有一次，源站故障了，客户反馈说，它们接入的多家CDN厂商都上报，文件下载失败了，只有我们公司文件下载成功了，听到这消息，我嘴角不由得上翘，漏出了得意的笑容。\u003C\u002Fp>\n\n\u003Ch3>并行写同文件\u003C\u002Fh3>\n\n\u003Cp>遇到另一个印象深刻的问题就是某客户反馈文件错误的问题。客户反馈说他们在我们CDN上的文件播到一半卡死了，查了文件之后，发现文件大小正常，但是文件只有前半截有数据，后半截全是0，空的。当时，老板CTO领导全站在我工位后面，讨论这个问题，我也是第一时间加回了MD5校验，因为之前在我同事的要求下（同事是老员工、他负责任务调度、我负责几个agent，相当于他为主），去掉了上传前MD5校验，以节省任务时间。当时非常紧张，但是一时间也查不出原因，只能先加MD5校验，然后开发专用校验程序，将半年前开始的所有上传文件重新拉下来做MD5校验，查出异常文件重新上传。后来在一个偶然的机会下，发现了故障的原因，居然是调度测下发任务之后，等待N分钟之后agent还没上报成功，于是又重新下发了这个任务，然后刚好是同一个节点的agent收到该任务，这样一来，同一个节点上2个相同的任务在处理，文件路径一致，也就是说，有2个线程在从源站拉文件写到本地磁盘的同一个路径，一先一后，先处理完的线程开始上传了，后处理的还在写文件，后处理的写文件之后就会将该文件后半截变为空（这里与文件打开模式有关），这样前一个线程在上传的时候，文件其实变成半截了，再加上没有MD5校验，所以，故障就此发生了。\u003C\u002Fp>\n\n\u003Cp>所以之后我就给每个任务的本地路径前加了一个uuid目录，这样一来，不管任务是否同一个文件路径，都不会相互干扰。每个任务都有自己的独立路径了。原程序是我接手别人的，虽说这问题不是我引入，但是，平心而论换做当时的我，也是想不到这一点的。不过，吃一堑长一智，现在只要涉及到本地资源存储，我基本上都会给路径前添加uuid前缀。\u003C\u002Fp>\n",{"title":17,"list":18},"最热文章",[19,25,31,37,43,49,55,61,67,73],{"id":20,"title":21,"uri":22,"author":9,"created_at":23,"view_count":24},28,"docker容器目录与主机目录同步","\u002Farticle\u002Fdocker容器目录与主机目录同步","2014-12-31 12:34:08",15570,{"id":26,"title":27,"uri":28,"author":9,"created_at":29,"view_count":30},21,"beego中使用fis","\u002Farticle\u002Fbeego中使用fis","2014-12-03 10:47:56",9250,{"id":32,"title":33,"uri":34,"author":9,"created_at":35,"view_count":36},20,"用github登录，oauth开发","\u002Farticle\u002F用github登录，oauth开发","2014-11-23 13:33:37",9113,{"id":38,"title":39,"uri":40,"author":9,"created_at":41,"view_count":42},18,"windows服务注册工具nssm","\u002Farticle\u002Fwindows服务注册工具nssm","2014-10-31 21:34:39",9047,{"id":44,"title":45,"uri":46,"author":9,"created_at":47,"view_count":48},49,"用latex排版中文书籍","\u002Farticle\u002F用latex排版中文书籍","2016-10-31 11:12:02",8634,{"id":50,"title":51,"uri":52,"author":9,"created_at":53,"view_count":54},19,"texlive中文书签乱码","\u002Farticle\u002Ftexlive中文书签乱码","2014-11-15 02:54:01",8461,{"id":56,"title":57,"uri":58,"author":9,"created_at":59,"view_count":60},6,"Windows上的软件包管理器chocolatey","\u002Farticle\u002FWindows上的软件包管理器chocolatey","2014-08-10 15:55:22",8009,{"id":62,"title":63,"uri":64,"author":9,"created_at":65,"view_count":66},3,"利用开源和免费资源搭建CI环境","\u002Farticle\u002F利用开源和免费资源搭建CI环境","2014-07-04 16:54:58",7732,{"id":68,"title":69,"uri":70,"author":9,"created_at":71,"view_count":72},16,"php-fpm网站目录权限配置","\u002Farticle\u002Fphp-fpm网站目录权限配置","2014-10-18 01:36:10",7274,{"id":74,"title":75,"uri":76,"author":9,"created_at":77,"view_count":78},27,"sublime text 2插件开发，外部修改文件无法重新加载","\u002Farticle\u002Fsublime text 2插件开发，外部修改文件无法重新加载","2014-12-26 20:24:13",7054,{"title":80,"list":81},"按月归档",[82,86,89,92,95,98,101,105,109,112,115,118,122,125,128,132,135,138,141,145,148,151,154,157,160,163,166,169,172,175,178,181,184,187,190,193,196,199,202,205,209,212,215,218,221,224],{"title":83,"uri":84,"view_count":85},"2026年7月","\u002Farchive\u002F2026\u002F7\u002F1","[1]",{"title":87,"uri":88,"view_count":85},"2026年1月","\u002Farchive\u002F2026\u002F1\u002F1",{"title":90,"uri":91,"view_count":85},"2025年4月","\u002Farchive\u002F2025\u002F4\u002F1",{"title":93,"uri":94,"view_count":85},"2024年6月","\u002Farchive\u002F2024\u002F6\u002F1",{"title":96,"uri":97,"view_count":85},"2024年5月","\u002Farchive\u002F2024\u002F5\u002F1",{"title":99,"uri":100,"view_count":85},"2024年4月","\u002Farchive\u002F2024\u002F4\u002F1",{"title":102,"uri":103,"view_count":104},"2023年12月","\u002Farchive\u002F2023\u002F12\u002F1","[6]",{"title":106,"uri":107,"view_count":108},"2023年11月","\u002Farchive\u002F2023\u002F11\u002F1","[2]",{"title":110,"uri":111,"view_count":85},"2023年10月","\u002Farchive\u002F2023\u002F10\u002F1",{"title":113,"uri":114,"view_count":85},"2023年7月","\u002Farchive\u002F2023\u002F7\u002F1",{"title":116,"uri":117,"view_count":108},"2023年6月","\u002Farchive\u002F2023\u002F6\u002F1",{"title":119,"uri":120,"view_count":121},"2023年5月","\u002Farchive\u002F2023\u002F5\u002F1","[3]",{"title":123,"uri":124,"view_count":108},"2023年4月","\u002Farchive\u002F2023\u002F4\u002F1",{"title":126,"uri":127,"view_count":108},"2023年3月","\u002Farchive\u002F2023\u002F3\u002F1",{"title":129,"uri":130,"view_count":131},"2023年2月","\u002Farchive\u002F2023\u002F2\u002F1","[5]",{"title":133,"uri":134,"view_count":85},"2023年1月","\u002Farchive\u002F2023\u002F1\u002F1",{"title":136,"uri":137,"view_count":108},"2022年12月","\u002Farchive\u002F2022\u002F12\u002F1",{"title":139,"uri":140,"view_count":131},"2022年11月","\u002Farchive\u002F2022\u002F11\u002F1",{"title":142,"uri":143,"view_count":144},"2022年10月","\u002Farchive\u002F2022\u002F10\u002F1","[4]",{"title":146,"uri":147,"view_count":104},"2022年9月","\u002Farchive\u002F2022\u002F9\u002F1",{"title":149,"uri":150,"view_count":131},"2022年8月","\u002Farchive\u002F2022\u002F8\u002F1",{"title":152,"uri":153,"view_count":85},"2019年7月","\u002Farchive\u002F2019\u002F7\u002F1",{"title":155,"uri":156,"view_count":85},"2019年2月","\u002Farchive\u002F2019\u002F2\u002F1",{"title":158,"uri":159,"view_count":85},"2018年9月","\u002Farchive\u002F2018\u002F9\u002F1",{"title":161,"uri":162,"view_count":85},"2018年8月","\u002Farchive\u002F2018\u002F8\u002F1",{"title":164,"uri":165,"view_count":85},"2017年11月","\u002Farchive\u002F2017\u002F11\u002F1",{"title":167,"uri":168,"view_count":85},"2017年8月","\u002Farchive\u002F2017\u002F8\u002F1",{"title":170,"uri":171,"view_count":85},"2017年5月","\u002Farchive\u002F2017\u002F5\u002F1",{"title":173,"uri":174,"view_count":85},"2017年4月","\u002Farchive\u002F2017\u002F4\u002F1",{"title":176,"uri":177,"view_count":85},"2016年11月","\u002Farchive\u002F2016\u002F11\u002F1",{"title":179,"uri":180,"view_count":85},"2016年10月","\u002Farchive\u002F2016\u002F10\u002F1",{"title":182,"uri":183,"view_count":108},"2016年9月","\u002Farchive\u002F2016\u002F9\u002F1",{"title":185,"uri":186,"view_count":85},"2016年8月","\u002Farchive\u002F2016\u002F8\u002F1",{"title":188,"uri":189,"view_count":108},"2016年7月","\u002Farchive\u002F2016\u002F7\u002F1",{"title":191,"uri":192,"view_count":131},"2016年6月","\u002Farchive\u002F2016\u002F6\u002F1",{"title":194,"uri":195,"view_count":85},"2015年12月","\u002Farchive\u002F2015\u002F12\u002F1",{"title":197,"uri":198,"view_count":85},"2015年10月","\u002Farchive\u002F2015\u002F10\u002F1",{"title":200,"uri":201,"view_count":108},"2015年2月","\u002Farchive\u002F2015\u002F2\u002F1",{"title":203,"uri":204,"view_count":85},"2015年1月","\u002Farchive\u002F2015\u002F1\u002F1",{"title":206,"uri":207,"view_count":208},"2014年12月","\u002Farchive\u002F2014\u002F12\u002F1","[8]",{"title":210,"uri":211,"view_count":108},"2014年11月","\u002Farchive\u002F2014\u002F11\u002F1",{"title":213,"uri":214,"view_count":131},"2014年10月","\u002Farchive\u002F2014\u002F10\u002F1",{"title":216,"uri":217,"view_count":121},"2014年9月","\u002Farchive\u002F2014\u002F9\u002F1",{"title":219,"uri":220,"view_count":121},"2014年8月","\u002Farchive\u002F2014\u002F8\u002F1",{"title":222,"uri":223,"view_count":121},"2014年7月","\u002Farchive\u002F2014\u002F7\u002F1",{"title":225,"uri":226,"view_count":108},"2013年9月","\u002Farchive\u002F2013\u002F9\u002F1",{"url":228,"title":229},"https:\u002F\u002Fimage.duguying.net\u002F2022\u002F09\u002Fij6x8o4s.webp","老家 摄于2020年秋",1790062265779]