{"id":56024,"date":"2024-05-09T15:12:58","date_gmt":"2024-05-09T15:12:58","guid":{"rendered":"https:\/\/packetstormsecurity.com\/news\/view\/35857\/AWS-CloudQuarry-Digging-For-Secrets-In-Public-AMIs.html"},"modified":"2024-05-09T15:12:58","modified_gmt":"2024-05-09T15:12:58","slug":"aws-cloudquarry-digging-for-secrets-in-public-amis","status":"publish","type":"post","link":"https:\/\/www.threatshub.org\/blog\/aws-cloudquarry-digging-for-secrets-in-public-amis\/","title":{"rendered":"AWS CloudQuarry: Digging For Secrets In Public AMIs"},"content":{"rendered":"

Money, secrets and mass exploitation: This research unveils a quarry of sensitive data stored in public AMIs. Digging through each AMI we managed to collect 500 GB of credentials, private repositories, access keys and more. The present article is the detailed analysis of how we did it and what the data represents.<\/p>\n

We did a coordinated disclosure with AWS\u2019s security team before publishing this article.<\/p>\n

Researchers and article authors:<\/p>\n

Before you begin<\/h2>\n

The results and methodology of this research could have been presented in a short summary along with 1-2 diagrams. However, we decided to go for a lengthy article that will describe our methodology, our thinking, trade-offs and results.<\/p>\n

We invite you to challenge our approach and find aspects that we missed out or could have been improved. You can contact us through our LinkedIn profiles, through the comment section or through the Security Cafe media channels that you can find in the Contact page.<\/p>\n

TABLE OF CONTENTS<\/h2>\n

1. Introduction<\/h2>\n

1.1 Some words from AWS security team<\/h3>\n

Note that AWS-owned AMIs are not affected by this issue and no customer action is required for those AMIs.<\/p>\n

Part of publishing AMIs is indeed publishing something public and customers should take care to inspect them prior to publishing so that they know what is in fact being published. As a best practice, AMI creators should not include credentials, including AWS account credentials, in published AMIs.<\/p>\n

For more information about crafting secure, shareable AMIs, recommendations can be found here:<\/p>\n

We also recommend customers use AMIs owned and published by AWS, or by another Amazon verified provider. For more information about verified providers, review \u201cVerified provider\u201d in the User Guide for Linux Instances. https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/sharing-amis.html#verified-ami-provider<\/a><\/p>\n

1.2 Research Idea \u2013 Origins<\/h3>\n

Some time ago I was doing a typical cloud security configuration review for a client. While checking for misconfigurations in the EC2 dashboard, I saw for the first time not one, but many public AMIs.<\/p>\n

But what is an AMI? Well, AMI stands for Amazon Machine Image and if it\u2019s the first time you hear about them that doesn\u2019t mean you haven\u2019t used one before. You see, every time you launch an EC2 instance, that instance is always based on an AMI.<\/p>\n

And to put it simply, an AMI is like a preconfigured virtual machine. Imagine you want to start an instance that has a certain database technology already installed and ready to go. Chances are that someone already configured such an instance and then made that instance an AMI so that others can use it.<\/p>\n

EC2 Launch Instance \u2013 Dashboard select AMI<\/figcaption><\/figure>\n

But most likely you used a common Amazon Linux AMI from the AWS Marketplace. These AMIs are from verified publishers. They are, in a sense, official AMIs. And then you have the Community AMIs, which are AMIs published by anyone and verified by none.<\/p>\n

To come back to my story, this client had multiple public AMIs that anyone could have found and used to start an EC2 instance based on that image. Essentially, anyone on the internet could have gained access to whatever files, secrets, credentials, code the client configured at some point and published \u201caccidentally\u201d (we\u2019ll talk later about those quotes).<\/p>\n

When I asked the client if I can check what was inside the AMIs, they said no ????. After finishing the engagement I was left with a question: Did they accidentally expose sensitive data all these years through these AMIs? And this gave birth to a second, bigger question: Are there other companies exposing sensitive data through public AMIs?<\/p>\n

While I didn\u2019t have the chance to get an answer for the first question, I knew I could answer the second one. So, with the help of my colleague Matei Josephs, we aimed to answer with this research if \u201cAre there any public AMIs with sensitive data out there?<\/strong>\u201c.<\/p>\n

1.3 Relevant Details About AMIs<\/h3>\n

First off, AMIs are dependent on their region. An AMI made public (published) in eu-central-1 will not be available in us-west-2 or any other region for that matter. This means that you have to specify the proper region if you want to access a particular AMI.<\/p>\n

Example AMI response from AWS API<\/figcaption><\/figure>\n

The second thing is that AMIs are by default created as private and it\u2019s actually hard to make it public. You have to disable a setting at the account level, wait 10 minutes, go to the AMI you want to make public and when you try to make it public you have to type \u201cconfirm\u201d in a pop-up that tells you the implied security risks. That\u2019s why I said the AMIs are published \u201caccidentally\u201d with quotes, because as far as I can tell, there is no accident in doing with; maybe just lack of awareness.<\/p>\n

Deprecation details from https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ami-deprecate.html<\/a><\/figcaption><\/figure>\n

The third thing is that AMIs can be deprecated. Let\u2019s look at the images above. Apart from the highlighted fields, we have on the last line a property called \u201cDeprecationTime\u201d. AMIs can become deprecated. As a user, you can deprecate your AMIs whenever you want. And if you don\u2019t, AWS will automatically deprecate your image after two years from its creation. This behavior was introduced in 2021 and its result is that deprecated images will not be accessible from the web console anymore. However, you can still use and query deprecated AMIs. You just have to use the CLI for that.<\/p>\n

CLI options for \u201cec2 describe-images\u201d regarding deprecation<\/figcaption><\/figure>\n

1.4 Previous Work<\/h3>\n

Dolev Farhi did something similar as us, but Dolev scanned AMIs from a single AWS region and only based on certain keywords: https:\/\/blog.lethalbit.com\/hunting-for-sensitive-data-in-public-amazon-images-ami\/<\/a><\/p>\n

We wanted to scan all public AMIs, so our work completes Dolev\u2019s by covering the other 26+ AWS regions and the AMIs from Dolev\u2019s region that we\u2019re left out<\/p>\n

Ben Morris did a research close to what we are doing. He scanned public EBS volumes for secrets as he presented in his talk at DEF CON 27: https:\/\/www.youtube.com\/watch?v=HXM1rBk_wXs<\/a><\/p>\n

While in theory we are scanning the AMI\u2019s volumes, in our case the EBS volume is not public by default just because the AMI is public. So there is very little overlap (we estimate under 5%) between his research and ours.<\/p>\n

Having this said, we believe that our research gives a more complete view into this issue that was almost not explored from an offensive perspective at large scale.<\/p>\n

1.5 Rules & Objectives<\/h3>\n

We set out some rules from the beginning. The most important one, act in an ethical manner. Practically, this research didn\u2019t involve any hacking or breaking into systems. It only relied on data publicly available and set out by companies or individuals. However, we felt that acting ethically should be always on our mind.<\/p>\n

We also established a few objectives:<\/p>\n

    \n
  1. Dig for secrets in every public AMI across every AWS region<\/li>\n
  2. Have Fun<\/li>\n
  3. Responsible Disclosure<\/li>\n<\/ol>\n

    So we wanted to find out if there are secrets exposed in public AMIs by searching through all of them, have fun while doing it and responsible disclose to issue to the affected companies if it\u2019s the case.<\/p>\n

    Let\u2019s get into it!<\/p>\n

    2. Collecting all Public AMIs<\/h2>\n

    First off, we enabled all AWS regions inside our AWS account. This was an easy process that didn\u2019t cost anything.<\/p>\n

    Now: the command for retrieving all public AMIs from a region is fairly simple:<\/p>\n

    \n
    \n# this will work for most regions with a default page size of 1000\naws ec2 describe-images --include-deprecated --region $region > $region.json # set a smaller page size if you get timeout errors\naws ec2 describe-images --include-deprecated --region $region --page-size 200 > $region.json\n<\/pre>\n<\/div>\n
    We knew this would be a challenging research when we had to wait a few hours to get all the public AMIs. Collecting the AMIs from every AWS region resulted in 11 GB of data ????<\/p>\n
    This was the point where we decided to give a name to the research and create a private repository. In Romanian we have this expression of \u201cdigging for something\u201d when you search for something. And while our objective was to dig for secrets in every public AMI, it simply clicked for us. That\u2019s how we got to the whole \u201cCloudQuarry\u201d and \u201cdigging\u201d thing.<\/p>\n
    \n
    CloudQuarry image generated by DALL\u00b7E 3<\/figcaption><\/figure>\n<\/div>\n
    We transformed the 11 GB into 750 MB of archives which we uploaded in our repository.<\/p>\n
    3. Processing the Collected AMIs<\/h2>\n
    3.1 Making sense of what we have<\/h3>\n
    We got a total of 3.1 million public AMIs. This number represents all the public AMIs from 27 AWS region (now there are more regions in AWS) available in December 2023 (more on the new public AMIs from that date at the end of the article).<\/p>\n
    Here are the statistics per region, ordered by count, at the start of the research:<\/p>\n
    \n
    \nus-east-1: 203,864\nus-west-2: 183,509\neu-west-1: 171,196\nus-west-1: 170,634\nap-southeast-1: 164,231\nap-northeast-1: 163,020\nap-southeast-2: 162,493\neu-central-1: 161,535\nsa-east-1: 156,402\nus-east-2: 147,191\nap-northeast-2: 142,417\neu-west-2: 141,971\nap-south-1: 141,404\nca-central-1: 137,669\neu-west-3: 131,319\neu-north-1: 120,526\nap-east-1: 104,742\nme-south-1: 100,369\neu-south-1: 96,996\naf-south-1: 93,072\nap-southeast-3: 61,353\nme-central-1: 44,236\neu-central-2: 41,351\neu-south-2: 40,912\nap-south-2: 37,096\nap-southeast-4: 34,652\nil-central-1: 19,072\nTotal images: 3,173,232\n<\/pre>\n<\/div>\n
    Now we faced a new question: is there any point in scanning all 3.1 million AMIs? We decided to further analyze the data to answer this. So we ran some python scripts to check things like how many unique owners are and top 10 AWS accounts with AMIs from each region. You can find the stats in the next file:<\/p>\n
    Just looking at the data above we can see that there are a few AWS accounts with tens of thousands of AMIs. We can do some google dorking on them to check who is the company behind them, but the idea is that most likely these big accounts are commercial publishers and if they are commercial publishers then the chances to have valid secrets in them are really low.<\/p>\n
    At this point we felt the need of a more easier way to analyze the data, so we moved it to MongoDB.<\/p>\n
    MongoDB Compass local instance<\/figcaption><\/figure>\n
    Using the Schema feature from MongoDB Compass we gained more insight into what was going on. For example, we saw that from a sample of 1000 random AMIs, 41% of them are owned by a single AWS account. That\u2019s over 1 million AMIs if extrapolated.<\/p>\n
    MongoDB Compass Schema analysis<\/figcaption><\/figure>\n
    3.2 Filtering out AMIs<\/h3>\n
    3.2.1 AMIs from Marketplace<\/h4>\n
    We decided that we can filter out AMIs based on some common sense conditions. The most obvious one was to filter out AMIs that are from the AWS Marketplace. The rationale behind this is that AMIs from marketplace are used on a daily basis by anyone. So, the chances to find valid secrets are small. Even more, these AMIs are from official and verified providers that are intentionally publish in the AWS Marketplace. We assume that they should be aware of what they publish and they most likely will not include sensitive information in their AMIs.<\/p>\n
    \n
    JSON Property for AMIs from AWS Marketplace<\/figcaption><\/figure>\n<\/div>\n
    We saw that the AMIs published in AWS Marketplace have an additional property that is not present for the community AMIs: ProductCodes<\/strong><\/p>\n
    To make our life easier, when we added the AMIs in MongoDB, we replaced the \u201cProductCodes\u201d field with a custom property called \u201cIsInMarketplace\u201d.<\/p>\n
    \n
    \ndef initialize_collection(amis_collection): for region in regions: amis = get_amis_for_region(region) for ami in amis: ami['Region'] = region ami['IsInMarketplace'] = 'ProductCodes' in ami.keys() ami['IsRootInstanceCoreType'] = ami['RootDeviceType'] == 'instance-store' amis_collection.insert_one(ami) print(f\"[x] Finished for region {region}\")\n<\/pre>\n<\/div>\n
    Doing this, we managed to filter out 1.6 million AMIs, reducing the total number from 3.1 million to 1.5 million AMIs. In the next file is the new count for each region:<\/p>\n
    Along the next chapters we\u2019ll build a MongoDB query that will filter all the necessary AMIs. This is how it looks for now:<\/p>\n
    \n
    \n{IsInMarketplace: false}\n<\/pre>\n<\/div>\n
    MongoDB Compass filter AMIs from AWS Marketplace<\/figcaption><\/figure>\n
    3.2.3 AMIs owned by AWS<\/h4>\n
    Analyzing the fields available for the remaining AMIs, we noticed that there was only one value for the field ImageOwnerAlias<\/strong>, and that value was \u201camazon\u201d.<\/p>\n
    \n
    AMI owned by AWS<\/figcaption><\/figure>\n<\/div>\n
    So the AMIs owned by AWS can be identified through property with the value \u201camazon\u201d. Well, we again assumed that there is no point in searching for secrets in AWS owned AMIs. In the end, they know better. While that might not be the case, we assumed the best and decided to filter them out.<\/p>\n
    MongoDB query filter:<\/p>\n
    \n
    \n{IsInMarketplace:false, ImageOwnerAlias: {$not:{$eq:\"amazon\"}}}\n<\/pre>\n<\/div>\n
    MongoDB Compass filter AMIs owned by AWS<\/figcaption><\/figure>\n
    Hey! Look at that! We got the number down to 1 million AMIs. While our objective was not to reduce the number of AMIs, but rather to construct a list of high potential targets, seeing a more feasible number of AMIs gives us the hope that we can scan all the AMIs worth scanning.<\/p>\n
    3.2.3 AMIs with RootDeviceType set to \u201cinstance-store\u201d<\/h4>\n
    Doing some tests we noticed that we can\u2019t start or access AMIs that have the property RootDeviceType<\/strong> set to \u201cinstance-store\u201d. These types of AMIs have the next characteristic:<\/p>\n
    \n
    Amazon instance store-backed AMI \u2013 The root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3.<\/p>\n
    https:\/\/docs.aws.amazon.com\/AWSEC2\/latest\/UserGuide\/ComponentsAMIs.html#storage-for-the-root-device<\/a><\/cite><\/p><\/blockquote>\n
    So we can\u2019t access these AMIs without the source from the S3 bucket. We decided to not scan these AMIs since we don\u2019t know where the root device is stored and since we won\u2019t have access most likely to that root device.<\/p>\n
    Again, we did the same thing when adding the AMIs in MongoDB, we changed the \u201cRootDeviceType\u201d property with a new field called \u201cIsRootInstanceCoreType\u201d. Now that I look back, I can see that the name should have been \u201cIsRootInstanceStore<\/strong>Type\u201d, but let\u2019s leave it as it is.<\/p>\n
    \n
    \ndef initialize_collection(amis_collection): for region in regions: amis = get_amis_for_region(region) for ami in amis: ami['Region'] = region ami['IsInMarketplace'] = 'ProductCodes' in ami.keys() ami['IsRootInstanceCoreType'] = ami['RootDeviceType'] == 'instance-store' amis_collection.insert_one(ami) print(f\"[x] Finished for region {region}\")\n<\/pre>\n<\/div>\n
    MongoDB query filter:<\/p>\n
    \n
    \n{IsInMarketplace:false, ImageOwnerAlias: {$not:{$eq:\"amazon\"}}, IsRootInstanceCoreType:false}\n<\/pre>\n<\/div>\n
    MongoDB Compass filter AMIs with RootDeviceType set to \u201cinstance-store\u201d<\/figcaption><\/figure>\n
    We only removed about 5k instances with this filter, but we managed to get below 1 million AMIs. The next filter however is the big game changer.<\/p>\n
    3.2.4 Owners with more than 50 Public AMIs<\/h4>\n
    Analyzing the remaining sample of AMIs, we noticed that we still have some AWS account with a large number of public AMIs.<\/p>\n
    MongoDB Compass analysis based on OwnerId<\/figcaption><\/figure>\n
    Checking the owners with the most AMIs we noticed that they are some kind of unofficial vendors, publishing new versions of their AMIs. Well, this is good, but how can we determine which account is some sort of vendor and which is not?<\/p>\n
    We were looking for a threshold. Something like, if a single AWS account owns more than X AMIs, then that AWS account is likely to be an unofficial vendor and we can exclude it. In the search of that threshold, we checked how many unique AWS accounts are out there with a maximum number of 25 public AMIs. Similarly, we did the same check for a maximum number of 20, 15, 10, 5 and 1. The next file contains the results:<\/p>\n
    Looking at this data we saw that numbers between the checked thresholds are fairly close. We factored in another thing. We assumed that no one is crazy enough to have more than 2 public AMIs on each AWS region. And imagine this: an AWS account has more than 50 public AMIs unintentionally left out there with source code, secrets, keys and so on. It felt like simply too much.<\/p>\n
    So, we introduced what I felt to be the first trade-off. We set this threshold of maximum 50 public AMIs per AWS account. Even after almost half a year later, I still think this was a good call. Doing some scripting in Python we identified those accounts with more than 50 AMIs. You can find the list in the MangoDB filter query below:<\/p>\n
    \n
    \n{ IsInMarketplace: false, IsRootInstanceCoreType: false, ImageOwnerAlias: {$not:{$eq:\"amazon\"}}, OwnerId: { $nin: [ \"979382823631\",\"125523088429\",\"782774275127\",\"085178109370\",\"462397596885\",\"504405067906\",\"782442783595\",\"575461648593\",\"281784684473\",\"126027368216\",\"958347557273\",\"531415883065\",\"357466774442\",\"118940168514\",\"172840064832\",\"364390758643\",\"108329106114\",\"161831738826\",\"958145178466\",\"258751437250\",\"138312800438\",\"939706979954\",\"211372476111\",\"992587258474\",\"099720109477\",\"281784684473\",\"679593333241\",\"953345546836\",\"014677528312\",\"531533671159\",\"275272412307\",\"150196106750\",\"692314432491\",\"297478128798\",\"609069431582\",\"659279834602\",\"304251891748\",\"074754820263\",\"082051669340\",\"083837402522\",\"240152784131\",\"012284807008\",\"833371238208\",\"781110569745\",\"852100819133\",\"666044672026\",\"040921035827\",\"641549025415\",\"602111481490\",\"470264848403\",\"164326021446\",\"263721492972\",\"677635134310\",\"604437731932\",\"651188399649\",\"117403053999\",\"273419675927\",\"909031101827\",\"675738832863\",\"973501904202\",\"078680276960\",\"101072000470\",\"601427279990\",\"847205733218\",\"033737991781\",\"867686887310\",\"515116089304\",\"253213882263\",\"922711891673\",\"107025636812\",\"144433228153\",\"361870911536\",\"981484854994\",\"981586224937\",\"341718127296\",\"012177264511\",\"506463145083\",\"343299325021\",\"798920643433\",\"672242615610\",\"193273892879\",\"088819544517\",\"619681448102\",\"955366579413\",\"213396452403\",\"833997227572\",\"696905082017\",\"481432522479\",\"633059600857\",\"298058684257\",\"872251088935\",\"558211695259\",\"717539892973\",\"761449685630\",\"458419666758\",\"117936542248\",\"262744063927\",\"156597721064\",\"700366075446\",\"228847291132\",\"929637706107\",\"375021575472\",\"221210582303\",\"221354433967\",\"811078382625\",\"604384805766\",\"747514752573\",\"268367799918\",\"002243364032\",\"903897969027\",\"391477561959\",\"634922670793\",\"691173103445\",\"121773538331\",\"894535543691\",\"167422901596\",\"897883143566\",\"808508158480\",\"178480506716\",\"961113262821\",\"128329325849\",\"649294524699\",\"799597332240\",\"857032533038\",\"701511253276\",\"902347396780\",\"342470128466\",\"063491364108\",\"056126556840\",\"823218754427\",\"622992712626\",\"776682305951\",\"524466471676\",\"522179138863\",\"066803512886\",\"332551323846\",\"623002322076\",\"005923036815\",\"725085649209\",\"159375186019\",\"830106197091\",\"055761336000\",\"153434714329\",\"805912557329\",\"231994803134\",\"785895325085\",\"664284451649\",\"643366669028\",\"120340763379\",\"860900021006\",\"811682917860\",\"030444862968\",\"734147683238\",\"314270375726\",\"743288062907\",\"411745053086\",\"920152662742\",\"113073460856\",\"511242591283\",\"158855661827\",\"003483243915\",\"107042026245\",\"308029823110\",\"222958650704\",\"489483686436\",\"275994140148\",\"212021838531\",\"879268495141\",\"208491357083\",\"666882590071\",\"226465381217\",\"062844181125\",\"703926772882\",\"940762080936\",\"872909225683\",\"085996170456\",\"266183714499\",\"023992783472\",\"812481205477\",\"644630763931\",\"766915741798\",\"922761411349\",\"958621391921\",\"471211731895\",\"567502172578\",\"517062167202\",\"576481180806\",\"422652172681\",\"233757773908\",\"753722448458\",\"930457884660\",\"220197662125\",\"858280671036\",\"413071817000\",\"754685078599\",\"260760892802\",\"919814621061\",\"370050660245\",\"672550935748\",\"988443417517\",\"174203920225\",\"959243851260\",\"391847209875\",\"654897662046\",\"261542671292\",\"514734236355\",\"787223699828\",\"079972220921\",\"879742242852\",\"455864098378\",\"092722883498\",\"187515304745\",\"354643243794\",\"020679151690\",\"641886270455\",\"367134611783\",\"529661837159\",\"769790836554\",\"354613140921\",\"398653132559\",\"609594173164\",\"679090434138\",\"143320424681\",\"887485515065\",\"943353741583\",\"745719312218\",\"514117492693\",\"072157914504\",\"963121779227\",\"657159205431\",\"999867407951\",\"160082703681\",\"302634334388\",\"153242070384\",\"492681118881\",\"646472328658\",\"441566271543\",\"223093067001\",\"568608671756\",\"811737349733\",\"670387110090\",\"614612213257\",\"012067661104\",\"183081753049\",\"555854959998\",\"372903911592\",\"931685398798\",\"711772050517\",\"397995763618\",\"626107101308\",\"188439194153\",\"328217207599\",\"303519801974\",\"297975325230\",\"488931611485\",\"013230744746\",\"125035497591\",\"331174399982\",\"275898483189\",\"732123782549\",\"636156223522\",\"210811600188\",\"835705546860\",\"902696655598\",\"381009627173\",\"887450378614\",\"505803097751\",\"121695370940\",\"159737981378\",\"465267886299\",\"034389035875\",\"055279482367\",\"502976614458\",\"851065383862\",\"843529240033\",\"481155165509\",\"542040705575\",\"009967377517\",\"105532983076\",\"077742499581\",\"263245908434\",\"887764444972\",\"049929217248\",\"705482507833\",\"883755651572\",\"036086565681\",\"400392161012\",\"674059360964\",\"174003430611\",\"108494951140\",\"491925185505\",\"178579023202\",\"689494258501\",\"966122790603\",\"324146450368\",\"648710915214\",\"687241535879\",\"042423532529\",\"299934467172\",\"772315742791\",\"582244788873\",\"084912087861\",\"601277729239\",\"210686502322\",\"456531263183\",\"142421590066\",\"410186602215\",\"253379484728\",\"942344369036\",\"090003032930\",\"390302536420\",\"694121914657\",\"344758251446\",\"733078701466\",\"746013851261\",\"273854932432\",\"608014515287\",\"900854079004\",\"919239153674\",\"388963328112\",\"463018831373\",\"760937637931\",\"511311637224\",\"897878824798\",\"887664104594\",\"687725076918\",\"813909747961\",\"141318035583\",\"068287425771\",\"317307795746\",\"948542851936\",\"105571261999\",\"034360980390\",\"001382923476\",\"054713022081\",\"861068367966\",\"558986605633\",\"541621974560\",\"478299826944\",\"873237481665\",\"865636555390\",\"154239132089\",\"709601304699\",\"579393713941\",\"833372943033\",\"625240141681\",\"701759196663\",\"699717368611\",\"704708324783\",\"130167558768\",\"270051122755\",\"415789037893\",\"235279919161\",\"893612263489\",\"098706035825\",\"560348900601\",\"549196663111\",\"635144173025\",\"971246865984\",\"764336703387\",\"904411313395\",\"801328867837\",\"227120241369\",\"371522382791\",\"816268476753\",\"247716482002\",\"142548018081\",\"817633883686\",\"578985338881\",\"649843420731\",\"617935088040\",\"328669466519\",\"160390125595\",\"320704736504\",\"701208020788\",\"995913728426\",\"312594956781\",\"022044324644\",\"582547558294\",\"155889952199\",\"346947735772\",\"428641199958\",\"675552985074\",\"849201093595\",\"080433136561\",\"511657745411\",\"148102829632\",\"044893506424\",\"434481986642\",\"838367653738\",\"000692753149\",\"216958472421\",\"233975207881\",\"007561666938\",\"895523100917\",\"466085125143\",\"821543836110\",\"763870024620\",\"589768463761\",\"131827586825\",\"851601128636\",\"322151761634\",\"700650335418\",\"572074891743\",\"723754913286\",\"119175775298\",\"795746500882\",\"917001008158\",\"068169053218\",\"792107900819\",\"536652696790\",\"976433692360\",\"515798882395\",\"483285841698\",\"942504563685\",\"965351925530\",\"544319711298\",\"600530653622\",\"748115360335\",\"940213080334\",\"362695845571\",\"686087431763\",\"086951212264\",\"688023202711\",\"706635527432\",\"728137396354\",\"895557238572\",\"496014204152\",\"200193376361\",\"089691119308\",\"459220251735\",\"166425443641\",\"573932760021\",\"605812595337\",\"383156758163\",\"890968705060\",\"406659825668\",\"710464970777\",\"734555027572\",\"072360076281\",\"379101102735\",\"400306079213\",\"020014417079\",\"800406105498\",\"921547910285\",\"057670693668\",\"214575254960\",\"643313122712\",\"807659967283\",\"793995986362\",\"411009282317\",\"055430919509\",\"149754531711\",\"766535289950\",\"137677339600\",\"282316784512\",\"474125479812\",\"519405898872\",\"330183209093\",\"262212597706\",\"910732127950\",\"143080556111\",\"538276064493\",\"915533145685\",\"562637147889\",\"840044800169\",\"762429196748\",\"867058893013\",\"657871693752\",\"174003592710\",\"421085264799\",\"181232990041\",\"374168611083\",\"540036508848\",\"042001279082\",\"855262394183\",\"952013314444\",\"666563790427\",\"146628656107\",\"161595045889\",\"067063543704\",\"101720206348\",\"075585003325\",\"185007729374\",\"980678866538\",\"867471893442\",\"797456418907\",\"903794441882\" ] } }\n<\/pre>\n<\/div>\n
    Well, this filter brought us to the more realistic number of 27,858<\/strong> AMIs.<\/p>\n
    MongoDB Compass filter owners with more than 50 public AMIs<\/figcaption><\/figure>\n
    So, we removed 488 AWS accounts and their associated public AMIs. This removed 970,481 AMIs. So, to put things in perspective, as a mean, each AWS account removed had about 2000 public AMIs.<\/p>\n
    Let\u2019s do some sanity checks and complete our list of targets.<\/p>\n
    3.2.5 Sanity Checks<\/h4>\n
    We are almost ready. The last step we did when filtering out instances was to set some limits in terms of volumes number and volumes size.<\/p>\n
    Firstly, each AMI comes with one or more EBS Snapshots. These are basically the volumes (disks) associated with the AMI. And when we start an EC2 instance based, we essentially get a copy of the AMI\u2019s volumes in our account.<\/p>\n
    When checking the remaining AMIs from MongoDB we noticed that a few AMIs had some extremely big volumes or a lot of volumes. This would have required a lot of scanning time, increased cost and overall a headache.<\/p>\n
    As an example, we encountered AMIs with 28 volumes or with volumes big as 20 TB of data. As my colleague Matei like to cite from the Zen of Python, special cases aren\u2019t special enough to break the rules.<\/p>\n
    \n
    Example of AMI with a volumes of 20 TB of data<\/figcaption><\/figure>\n<\/div>\n
    So, we checked some means and we decided to filter out any AMI that has more than 3 volumes or with volumes bigger than 200 GB.<\/p>\n
    \n
    \nfor ami in amis: bdm = ami['BlockDeviceMappings'] ebs_items = [x['Ebs'] for x in bdm if 'Ebs' in x.keys()] ebs_items = [x for x in ebs_items if 'SnapshotId' in x.keys()] if len(ebs_items) > 3: bad_amis.append(ami) continue big_snapshots = [x for x in ebs_items if x['VolumeSize'] > 200] if len(big_snapshots) > 0: bad_amis.append(ami)\n<\/pre>\n<\/div>\n
    We didn\u2019t know how to do this in MongoDB so we did some classic python scripting. After this final filtering we brought down the number of AMIs to 26,778<\/strong>.<\/p>\n
    And with this we managed to construct a list of targets. We were ready to start looking on ways to access the AMIs\u2019 contents \u2764\ufe0f<\/p>\n
    4. Accessing the AMIs<\/h2>\n
    For us it made sense to not look into ways of how to access the AMIs before we had a final number of AMIs. Mostly because based on how many AMIs there were, we had to prioritize either time, costs or scanning robustness.<\/p>\n
    With 26k AMIs, we knew we had to find an option that:<\/p>\n