[{"data":1,"prerenderedAt":232},["ShallowReactive",2],{"tag-下载-1":3},{"title":4,"articles":5,"hot_article":17,"month_arch":80,"total":228,"cover":229},"下载 - 独孤影的博客",[6],{"id":7,"title":8,"uri":9,"author":10,"tags":11,"created_at":14,"view_count":15,"content":16},69,"大文件下载的一些经验","how-to-download-big-files","duguying",[12,13],"下载","cdn","2022-08-26 15:32:21",494,"\u003Cp>数年之前，我在一家CDN公司上班。在CDN项目主要工作是负责对客户视频数据搬运到云存储，中间也涉及到下载转码上传之类的工作。其中踩过一些坑，也吸取了一些经验。下面简单讲一下我遇到的几个问题。\u003C\u002Fp>\n\n\u003Ch3>下载没网速了\u003C\u002Fh3>\n\n\u003Cp>首先是下载没网速的情况。在处理大量的下载任务过程中，发现了一些特殊情况，客户通知我的服务回源，我的下载服务从源站下载一些大文件的时候，开始网络一切正常，可是逐渐的网速越来越低，甚至没网速，这回导致下载一直卡主。可能你会说，设置一个下载超时不就好了，但是，由于客户的文件大多是影视原片，体积巨大，有些甚至达到了几十G，如果简单的设置超时，以处理时间判断下载成败，那么这些大文件基本上不可能成功，都会以超时失败。当然，也可以分片下载来解决文件大的问题，但是对于0网速或者只有几B\u002Fs的网速，你的分片就显得无力了。其实，这个状况产生的原因大多是因为源站故障导致的，比如负载高之类的。只要断开连接重新下载就有机会恢复正常速度的连接，而且一般源站是有多个的，重新连接到新的节点就好了。所以，这里“断开连接重新发起下载”这个能力就非常重要了。我原本使用的是\u003Ccode class=\"inline-code\">github.com\u002Fnareix\u002Fcurl\u003C\u002Fcode>这个库进行文件下载，它的优点是创建异步下载任务后，可以监控当前下载的速度。这样，我可以对每个下载任务的网速进行监控，当网速低于一个阈值的时候，就放弃下载并重试。最初的时候只是简单的调用它的\u003Ccode class=\"inline-code\">Close()\u003C\u002Fcode>方法，关闭了下载。后面却发现，经常发生fd用完的情况，一查才知道\u003Ccode class=\"inline-code\">Close()\u003C\u002Fcode>方法并不能释放连接。最后只好自己修改了这个库，给它添加了强制关闭连接的能力。于是就完美的解决了这一问题。记得当时有一次，源站故障了，客户反馈说，它们接入的多家CDN厂商都上报，文件下载失败了，只有我们公司文件下载成功了，听到这消息，我嘴角不由得上翘，漏出了得意的笑容。\u003C\u002Fp>\n\n\u003Ch3>并行写同文件\u003C\u002Fh3>\n\n\u003Cp>遇到另一个印象深刻的问题就是某客户反馈文件错误的问题。客户反馈说他们在我们CDN上的文件播到一半卡死了，查了文件之后，发现文件大小正常，但是文件只有前半截有数据，后半截全是0，空的。当时，老板CTO领导全站在我工位后面，讨论这个问题，我也是第一时间加回了MD5校验，因为之前在我同事的要求下（同事是老员工、他负责任务调度、我负责几个agent，相当于他为主），去掉了上传前MD5校验，以节省任务时间。当时非常紧张，但是一时间也查不出原因，只能先加MD5校验，然后开发专用校验程序，将半年前开始的所有上传文件重新拉下来做MD5校验，查出异常文件重新上传。后来在一个偶然的机会下，发现了故障的原因，居然是调度测下发任务之后，等待N分钟之后agent还没上报成功，于是又重新下发了这个任务，然后刚好是同一个节点的agent收到该任务，这样一来，同一个节点上2个相同的任务在处理，文件路径一致，也就是说，有2个线程在从源站拉文件写到本地磁盘的同一个路径，一先一后，先处理完的线程开始上传了，后处理的还在写文件，后处理的写文件之后就会将该文件后半截变为空（这里与文件打开模式有关），这样前一个线程在上传的时候，文件其实变成半截了，再加上没有MD5校验，所以，故障就此发生了。\u003C\u002Fp>\n\n\u003Cp>所以之后我就给每个任务的本地路径前加了一个uuid目录，这样一来，不管任务是否同一个文件路径，都不会相互干扰。每个任务都有自己的独立路径了。原程序是我接手别人的，虽说这问题不是我引入，但是，平心而论换做当时的我，也是想不到这一点的。不过，吃一堑长一智，现在只要涉及到本地资源存储，我基本上都会给路径前添加uuid前缀。\u003C\u002Fp>\n",{"title":18,"list":19},"最热文章",[20,26,32,38,44,50,56,62,68,74],{"id":21,"title":22,"uri":23,"author":10,"created_at":24,"view_count":25},28,"docker容器目录与主机目录同步","\u002Farticle\u002Fdocker容器目录与主机目录同步","2014-12-31 12:34:08",15573,{"id":27,"title":28,"uri":29,"author":10,"created_at":30,"view_count":31},21,"beego中使用fis","\u002Farticle\u002Fbeego中使用fis","2014-12-03 10:47:56",9253,{"id":33,"title":34,"uri":35,"author":10,"created_at":36,"view_count":37},20,"用github登录，oauth开发","\u002Farticle\u002F用github登录，oauth开发","2014-11-23 13:33:37",9115,{"id":39,"title":40,"uri":41,"author":10,"created_at":42,"view_count":43},18,"windows服务注册工具nssm","\u002Farticle\u002Fwindows服务注册工具nssm","2014-10-31 21:34:39",9054,{"id":45,"title":46,"uri":47,"author":10,"created_at":48,"view_count":49},49,"用latex排版中文书籍","\u002Farticle\u002F用latex排版中文书籍","2016-10-31 11:12:02",8645,{"id":51,"title":52,"uri":53,"author":10,"created_at":54,"view_count":55},19,"texlive中文书签乱码","\u002Farticle\u002Ftexlive中文书签乱码","2014-11-15 02:54:01",8466,{"id":57,"title":58,"uri":59,"author":10,"created_at":60,"view_count":61},6,"Windows上的软件包管理器chocolatey","\u002Farticle\u002FWindows上的软件包管理器chocolatey","2014-08-10 15:55:22",8012,{"id":63,"title":64,"uri":65,"author":10,"created_at":66,"view_count":67},3,"利用开源和免费资源搭建CI环境","\u002Farticle\u002F利用开源和免费资源搭建CI环境","2014-07-04 16:54:58",7740,{"id":69,"title":70,"uri":71,"author":10,"created_at":72,"view_count":73},16,"php-fpm网站目录权限配置","\u002Farticle\u002Fphp-fpm网站目录权限配置","2014-10-18 01:36:10",7276,{"id":75,"title":76,"uri":77,"author":10,"created_at":78,"view_count":79},27,"sublime text 2插件开发，外部修改文件无法重新加载","\u002Farticle\u002Fsublime text 2插件开发，外部修改文件无法重新加载","2014-12-26 20:24:13",7055,{"title":81,"list":82},"按月归档",[83,87,90,93,96,99,102,106,110,113,116,119,123,126,129,133,136,139,142,146,149,152,155,158,161,164,167,170,173,176,179,182,185,188,191,194,197,200,203,206,210,213,216,219,222,225],{"title":84,"uri":85,"view_count":86},"2026年7月","\u002Farchive\u002F2026\u002F7\u002F1","[1]",{"title":88,"uri":89,"view_count":86},"2026年1月","\u002Farchive\u002F2026\u002F1\u002F1",{"title":91,"uri":92,"view_count":86},"2025年4月","\u002Farchive\u002F2025\u002F4\u002F1",{"title":94,"uri":95,"view_count":86},"2024年6月","\u002Farchive\u002F2024\u002F6\u002F1",{"title":97,"uri":98,"view_count":86},"2024年5月","\u002Farchive\u002F2024\u002F5\u002F1",{"title":100,"uri":101,"view_count":86},"2024年4月","\u002Farchive\u002F2024\u002F4\u002F1",{"title":103,"uri":104,"view_count":105},"2023年12月","\u002Farchive\u002F2023\u002F12\u002F1","[6]",{"title":107,"uri":108,"view_count":109},"2023年11月","\u002Farchive\u002F2023\u002F11\u002F1","[2]",{"title":111,"uri":112,"view_count":86},"2023年10月","\u002Farchive\u002F2023\u002F10\u002F1",{"title":114,"uri":115,"view_count":86},"2023年7月","\u002Farchive\u002F2023\u002F7\u002F1",{"title":117,"uri":118,"view_count":109},"2023年6月","\u002Farchive\u002F2023\u002F6\u002F1",{"title":120,"uri":121,"view_count":122},"2023年5月","\u002Farchive\u002F2023\u002F5\u002F1","[3]",{"title":124,"uri":125,"view_count":109},"2023年4月","\u002Farchive\u002F2023\u002F4\u002F1",{"title":127,"uri":128,"view_count":109},"2023年3月","\u002Farchive\u002F2023\u002F3\u002F1",{"title":130,"uri":131,"view_count":132},"2023年2月","\u002Farchive\u002F2023\u002F2\u002F1","[5]",{"title":134,"uri":135,"view_count":86},"2023年1月","\u002Farchive\u002F2023\u002F1\u002F1",{"title":137,"uri":138,"view_count":109},"2022年12月","\u002Farchive\u002F2022\u002F12\u002F1",{"title":140,"uri":141,"view_count":132},"2022年11月","\u002Farchive\u002F2022\u002F11\u002F1",{"title":143,"uri":144,"view_count":145},"2022年10月","\u002Farchive\u002F2022\u002F10\u002F1","[4]",{"title":147,"uri":148,"view_count":105},"2022年9月","\u002Farchive\u002F2022\u002F9\u002F1",{"title":150,"uri":151,"view_count":132},"2022年8月","\u002Farchive\u002F2022\u002F8\u002F1",{"title":153,"uri":154,"view_count":86},"2019年7月","\u002Farchive\u002F2019\u002F7\u002F1",{"title":156,"uri":157,"view_count":86},"2019年2月","\u002Farchive\u002F2019\u002F2\u002F1",{"title":159,"uri":160,"view_count":86},"2018年9月","\u002Farchive\u002F2018\u002F9\u002F1",{"title":162,"uri":163,"view_count":86},"2018年8月","\u002Farchive\u002F2018\u002F8\u002F1",{"title":165,"uri":166,"view_count":86},"2017年11月","\u002Farchive\u002F2017\u002F11\u002F1",{"title":168,"uri":169,"view_count":86},"2017年8月","\u002Farchive\u002F2017\u002F8\u002F1",{"title":171,"uri":172,"view_count":86},"2017年5月","\u002Farchive\u002F2017\u002F5\u002F1",{"title":174,"uri":175,"view_count":86},"2017年4月","\u002Farchive\u002F2017\u002F4\u002F1",{"title":177,"uri":178,"view_count":86},"2016年11月","\u002Farchive\u002F2016\u002F11\u002F1",{"title":180,"uri":181,"view_count":86},"2016年10月","\u002Farchive\u002F2016\u002F10\u002F1",{"title":183,"uri":184,"view_count":109},"2016年9月","\u002Farchive\u002F2016\u002F9\u002F1",{"title":186,"uri":187,"view_count":86},"2016年8月","\u002Farchive\u002F2016\u002F8\u002F1",{"title":189,"uri":190,"view_count":109},"2016年7月","\u002Farchive\u002F2016\u002F7\u002F1",{"title":192,"uri":193,"view_count":132},"2016年6月","\u002Farchive\u002F2016\u002F6\u002F1",{"title":195,"uri":196,"view_count":86},"2015年12月","\u002Farchive\u002F2015\u002F12\u002F1",{"title":198,"uri":199,"view_count":86},"2015年10月","\u002Farchive\u002F2015\u002F10\u002F1",{"title":201,"uri":202,"view_count":109},"2015年2月","\u002Farchive\u002F2015\u002F2\u002F1",{"title":204,"uri":205,"view_count":86},"2015年1月","\u002Farchive\u002F2015\u002F1\u002F1",{"title":207,"uri":208,"view_count":209},"2014年12月","\u002Farchive\u002F2014\u002F12\u002F1","[8]",{"title":211,"uri":212,"view_count":109},"2014年11月","\u002Farchive\u002F2014\u002F11\u002F1",{"title":214,"uri":215,"view_count":132},"2014年10月","\u002Farchive\u002F2014\u002F10\u002F1",{"title":217,"uri":218,"view_count":122},"2014年9月","\u002Farchive\u002F2014\u002F9\u002F1",{"title":220,"uri":221,"view_count":122},"2014年8月","\u002Farchive\u002F2014\u002F8\u002F1",{"title":223,"uri":224,"view_count":122},"2014年7月","\u002Farchive\u002F2014\u002F7\u002F1",{"title":226,"uri":227,"view_count":109},"2013年9月","\u002Farchive\u002F2013\u002F9\u002F1",1,{"url":230,"title":231},"https:\u002F\u002Fimage.duguying.net\u002F2022\u002F09\u002Fij6x8o4s.webp","老家 摄于2020年秋",1790781596002]