[{"data":1,"prerenderedAt":53},["ShallowReactive",2],{"$f3sg4j26nqh52k":3,"recommended":22},{"title":4,"path":5,"date":6,"updated":7,"tags":8,"excerpt":11,"readingMinutes":12,"commentCount":13,"html":14,"comments":15,"next":16,"prev":19},"S3 boto3 'StreamingBody' object has no attribute 'tell'","\u002F2018\u002F01\u002Fs3-boto3-streamingbody-object-has-no\u002F","2018-01-14T21:13:00.002Z","2018-01-15T08:21:15.041Z",[9,10],"how-to","programming","I was recently trying to work with the python package warcio and feeding an s3 object from the common crawl bucket directly into it. r = s3.get_object(Key='crawl-data\u002Ffile....', Bucket='commoncrawl') for record in…",1,0,"\u003Cp>I was recently trying to work with the python package \u003Cem>warcio\u003C\u002Fem> and feeding an s3 object from the common crawl bucket directly into it.\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cblockquote>\n\u003Cp>r = s3.get_object(Key='crawl-data\u002Ffile....', Bucket='commoncrawl')\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cp>for record in ArchiveIterator(r['Body']):\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>pass\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003C\u002Fblockquote>\n\u003Cp>However, this fails with the error:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cblockquote>\n\u003Cp>self.offset = self.fh.tell()\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cp>AttributeError: 'StreamingBody' object has no attribute 'tell'\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003C\u002Fblockquote>\n\u003Cp>The reason is that boto3 s3 objects don't support \u003Cem>tell\u003C\u002Fem>. It's easily fixable by creating a tiny class:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cblockquote>\n\u003Cp>class S3ObjectWithTell:\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>def \\_\\_init\\_\\_(self, s3object):\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    self.s3object = s3object\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    self.offset = 0\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>def read(self, amount=None):\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    result = self.s3object.read(amount)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    self.offset += len(result)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    return result\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>def close(self):\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    self.s3object.close()\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>def tell(self):\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003Cblockquote>\n\u003Cpre>\u003Ccode>    return self.offset\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003C\u002Fblockquote>\n\u003C\u002Fblockquote>\n\u003Cp>You can now use this class and change\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cblockquote>\n\u003Cp>for record in ArchiveIterator(r['Body']):\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003C\u002Fblockquote>\n\u003Cp>into\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cblockquote>\n\u003Cp>for record in ArchiveIterator(S3ObjectWithTell(r['Body'])):\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003C\u002Fblockquote>\n",[],{"title":17,"path":18},"Every Day Shampoo","\u002F2018\u002F10\u002Fevery-day-shampoo\u002F",{"title":20,"path":21},"Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS","\u002F2016\u002F01\u002Fparsing-10tb-of-metadata-26m-domains\u002F",[23,28,33,36,40,43,46,50],{"title":24,"path":25,"note":26,"series":27},"Mapping out Dark Code","\u002F2026\u002F03\u002Fmapping-out-dark-code\u002F","2026",false,{"title":29,"path":30,"note":31,"series":32},"The software that I love","\u002F2022\u002F10\u002Fthe-software-that-i-love\u002F","Series · 36 posts",true,{"title":34,"path":35,"note":26,"series":27},"Navigating the Fog of War: Age of Empires, AI Agents, and the Rise of Dark Code","\u002F2026\u002F05\u002Fnavigating-fog-of-war-age-of-empires-ai\u002F",{"title":37,"path":38,"note":39,"series":27},"The Future of Programming Systems - four thoughts","\u002F2024\u002F09\u002Fthe-future-of-programming-systems-four\u002F","2024",{"title":41,"path":42,"note":39,"series":27},"Always Optimize the Feedback Loop","\u002F2024\u002F07\u002Falways-optimize-feedback-loop\u002F",{"title":44,"path":45,"note":39,"series":27},"\"Security Is Our Top Priority\" is BS","\u002F2024\u002F07\u002Fsecurity-is-our-top-priority-is-bs\u002F",{"title":47,"path":48,"note":49,"series":27},"10x teams","\u002F2025\u002F02\u002F10x-teams\u002F","2025",{"title":51,"path":52,"note":49,"series":27},"Hiring, from the Other Side","\u002F2025\u002F10\u002Fhiring-from-other-side\u002F",1792876809102]