In this post, I’d like to highlight a couple of eye-catching announcements (12 to be a bit more precise) made throughout re:Invent 2018, AWS’s annual main event, which took place in Las Vegas, Nevada. I have never been to one of their conferences personally, but man, would I like to go someday. AWS has always been and still is the Nr 1 in Cloud services and solutions worldwide. Of course, Microsoft is catching up quickly. Let’s dig in and see what they’ve been up to.
AWS Outposts
By many, this is thought of as one of the biggest announcements made during re:Invent. AWS Outposts brings native AWS services, infrastructure, and operating models (it’s still unclear which services exactly) to virtually any data center, co-location space, or on-premises facility.
This will include the ability to use the same AOI’s, tools, hardware, and functionalities across on-premises and Cloud. In other words, their aiming for the optimal hybrid experience.
You have a choice between VMware Cloud on AWS Outposts – read more here – or the AWS native variant, which I’ve pretty much summed up in the paragraph above.
Earth chocking? I don’t think so, but a very welcome (and pretty huge) addition to their portfolio. Again, one of the main players (the biggest in Cloud) who recognizes that we live in a hybrid world – though I guess that’s old news by now.
The Outposts announcement fits in nicely with earlier releases of AWS Cloud-based services becoming available for on-premises use. like AWS RDS (database), and additional compute and GPU capabilities being added to their Snowball (Edge) product line on an almost ongoing basis.
Outposts is scheduled for GA in Q4 of 2019, which is still a long way out. Hopefully, more details will be shared soon.
Snowball Edge GPU
Snowball physical appliances (see the image below) are meant to transfer huge amounts of data from your on-premises datacenter to the AWS Cloud, secure, simple, and fast. Simply because it would take far too long to transfer all data across the internet, regardless of any additional technologies/solutions that could be used.
Next to the AWS Snowball they later released the AWS Snowball Edge, which includes compute resources including the ability to process data locally on the Snowball device (it spins up one or multiple EC2 machines for that) before sending it back to AWS. Again, bringing AWS Cloud compute resources to your own (local) datacenter.
Today, AWS offers two more options or extensions on the above mentioned Snowball Edge. According to Jeff Barr “Snowball Edge Compute Optimized and Snowball Edge Compute Optimized with GPU (the original Snowball Edge is now called Snowball Edge Storage Optimized). Both options include 42 TB of S3-compatible storage and 7.68 TB of NVMe SSD storage, and allow you to run any combination of instances that consume up to 52 vCPUs and 208 GiB of memory”
Here’s a picture of the two machines next to each other. As you can see, the Storage Optimized version (the ‘old’ one) is just a little smaller. Credits go to Jeff Barr who posted these pics earlier.
An overview of the EC2 instance options available:
AWS DataSync
As mentioned, many customers start their Cloud journey coming from, or combining it with an on-premises environment. As such, large amounts of data need to be transferred or synchronized to and from the AWS Cloud. Next to a bunch of other data transfer solutions available, (by the way, make sure to check out the Snowmobile as well, it’s pretty awesome), AWS DataSync provides us with up to 10 times faster transfer speeds when compared to other open source solutions.
It is based on a purpose-built data transfer protocol. Easy to set up and fully managed with virtually unlimited scalability options. It can be used to move, or I should say synchronize data into, or out of AWS, as well as to and from on-premises environments. Go here for some more detailed information.
Other AWS data transfer/sync solutions include AWS Snowball, AWS Snowmobile, Kinesis Data Firehose, S3 Transfer Acceleration, and AWS Storage Gateway.
AWS well-architected tool
It is based on the AWS Well Architected Framework, which is made up out of five main pillars (Operational Excellence, Security, Reliability, Performance Efficiency, and Cost Optimization), various core strategies and best practices for architecting systems in the cloud. It’s used by AWS and their Solutions Architects to perform Well-architected (as AWS likes to call them) reviews on customer infrastructures throughout the year – thousands of them.
The Well Architected Tool is a self-service tool. This way customers can assess and evaluate their own AWS environments and workloads to further fine-tune, apply best practices where applicable (as part of the five pillars mentioned earlier) and perhaps to save some money along the way.
More information can be found here.
Hibernation for EC2 instances
Next to new network optimized instance types, which also got announced here you now have the ability to hibernate certain EC2 machines, after boot and configuration. When a hibernate command is given the in-memory state of the machine will be written to a file stored on the root EBS volume, before shutting down – so at least compute wise, it won’t cost you any money.
Today, this feature is available for Amazon Linux 1 machines – M3, M4, M5, C3, C4, C5, R3, R4, and R5 instances. Currently, support for Linux 2 machines is being worked on. No word on Windows (just) yet. Before getting started, there are some things to consider, have a read here.
FSx for Windows File Server and Lustre
Windows applications often need Windows-based file storage. With Amazon FSx, you can launch highly durable and available Windows file systems that can be accessed from up to thousands of compute instances using the industry-standard SMB protocol (2.0 to 3.1.1). SSD storage is used to provide sub-milliseconds latency and more than enough IOPS to service thousands of machines at the same time.
It works with Microsoft’s Active Directory and uses DFS replication to support multi Availability Zones deployments. All Windows versions from Server 2008, Windows 7 and upwards are supported. Also, all current versions of Linux are supported. Since it is a fully managed service you’ll be up and running in minutes and won’t have to worry about pathing, maintenance and so on. Click here for more information.
The same service exists for Lustre, a popular open-source parallel file system that is designed for compute-intensive workloads. Same rules apply here as well, Amazon FSx can scale to hundreds of gigabytes per second (throughput), millions of IOPS, and it’s offered as a fully managed service.
Firecracker
A new virtualization technology based on KVM. Lightweight virtual machines, called Micro VM’s are launched within a fraction of a second. It combines the security and workload isolation of traditional virtual machines and the resource efficiency introduced with containers. However, this technology isn’t new to AWS. For some time now (I don’t how long exactly) it is being used to power AWS services like Lambda and Fargate.
Reading about potential launch times of 125 ms and around 5MiB of memory consumption per Micro VM, which is quite impressive, I can imagine that this will again provide a boost for devops in general. It’s fully open source, you’ll find it here on GitHub.
S3 intelligent Tiering
Throughout the last couple of months I’ve been studying AWS (successfully sat two exams along the way). Throughout the various courses/lectures that I have been watching/listening to/reading, I learned a great deal about the various storage options available within AWS, amongst other things. For those of you that don’t know, S3 is one of the main (object-based) storage options available as part of AWS. The S3 units that you configure are called ‘Buckets’, and they have been around for over 12,5 years now.
They’re used for data archiving purposes, backup and restore, DR scenario’s, big data (lake) solutions, and more. Currently, these are the options to choose from:
- Standard– Designed for frequently accessed data.
- Standard-IA– Designed for long-lived, infrequently accessed data.
- One Zone-IA– Designed for long-lived, infrequently accessed, non-critical data.
- Glacier– Designed for long-lived, infrequent accessed, archived critical data.
- New options are on the way.
Besides the above, S3 Buckets also come with optional lifecycle policies for you to configure. Using lifecycle policies you can tell S3 to transfer data from one type of S3 Bucket to another type of S3 Bucket after a certain amount of time – 30 days by default. Form Standard, to Standard-IA, for example, saving you money.
There is a so-called Storage Class Analytics tool available to help you identify which objects might be suitable to move to a, perhaps lower and thus cheaper class of storage tier. Do note that this is (still) a manual process and takes time – enter S3 intelligent Tiering.
S3 intelligent Tiering takes care of moving data between S3 buckets automatically, back and forth. It does this by closely monitoring the access patterns of your data. What this means is, when data is infrequently accessed (left alone for 30 days) it will be moved from a ‘frequent access’ tier to an ‘infrequent access’ tier.
The other way around is also true. When the system notices that data which has been moved to an ‘infrequent access’ tier is being accessed again it will automatically be moved back to the, or a ‘frequent access’ tier. With S3 intelligent Tiering there is no change in availability and durability numbers, latency included. Pretty neat stuff and something they will continue to build on in the (near) future, I’m sure.
EFS infrequent access storage
Amazon Elastic File System (Amazon EFS) provides a (simple, scalable, etc.) elastic file system for Linux-based workloads for use with AWS Cloud services and on-premises resources. It scales on demand and is offered as a fully managed service, go here for more detailed information.
With EFS infrequent access storage, files that have not been touched for 30 days can be moved to infrequent EFS storage automatically by use of lifecycle policies, just as with the S3 Buckets mentioned earlier. EFS Infrequent Access will be generally available in all regions where EFS is available as of early 2019.
Double EBS provisioned IOPS
AWS Block storage is provided by EBS (Elastic Block Storage), for use with EC2 instances. They come in two disk flavors, Solid State Drives (SSD) and Hard Disk Drives (HDD) – see the image below as well.
EBS Provisioned IOPS SSD (io1) is the most powerful of the 4, offering up to 32000 IOPS per volume, and 500 MB/s per volume, including 80000 max IOPS per instance when attached to an EC2 Nitro system.
At re:Invent they announced to double the number of IOPS as well as the 500 MB/s throughput per volume. Meaning that today the io1 SSD option offers up to 64000 IOPS per volume, a 1000 MB/s per volume, and 80000 max IOPS (unchanged) per instance, when attached to an EC2 Nitro system – smoking!
CloudWatch Log Insights
A powerful tool that spans almost all of AWS’s most important resources, services and applications. It is used to collect, monitor, and analyze various streams of data, and can take automated actions when certain thresholds have been reached or breached, for example.
CloudWatch collects logs from over 30 different AWS services, these include but are not limited to VPC Flow logs, Route53, Amazon API Gateway, AWS Lambda, AWS CloudTrail, and more. Metrics from distributed applications can be collected from over 70 AWS services like EC2, DynamoDB, S3, ECS, Lambda, and many others, including optional custom metrics.
Optionally you can build custom dashboards to display what is going on within your Cloud deployments, and set alarms from where automated actions can be triggered, like shutting down unused resources, for example, or trigger Auto Scaling Group actions, etc. Analyzing and correlating collected information is also easily done.
CloudWatch offers a data retention perios of up to 15 months enabling you to monitor trends with intervals of as little as 1-second. Log Analytics (Insights) allows you to drive actionable intelligence from your logs to address operational issues without needing to provision servers or manage software. This comes from the aws.amazon.com ‘You can instantly begin writing queries with aggregations, filters, and regular expressions. In addition, you can visualize time series data, drill down into individual log events, and export query results to CloudWatch Dashboards. This gives you complete operational visibility. With a few clicks in the AWS Management Console, you can start using Logs Insights to query logs sent to CloudWatch. You only pay for the queries you run.’
Arm based EC2 instances
Arm processors are produced by the ARM Holdings PLC. They’re also known as Reduced Instruction Set Computing (RISC) processors, able to produce high-performance quality even though the units themselves are of a much simpler design when compared to Intel processors, for example.
They are often used in mobile phones, tablets, gaming systems, and numerous other types of (smaller) electronic devices. They are cheap, low on power consumption, but still powerful enough to handle complex calculations. On the downside, they are not compatible with x86 based architectures, so no Windows on top. Intel processors consist out of more complex designs primarily meant to be compatible with all the different types of desktop and laptop computers out there.
On November the 26th AWS introduced their own Arm-based Gravitation processors. To go with it they also launched a series of A1 EC2 instances optimized for performance and costs, all fully based and built around the Gravitation Arm cores – see the overview below. My guess is that this is only the beginning.
All these machines are EBS optimized by default. I guess it goes without saying that these machines, and all other services and releases I have covered in this article will be, or are available in certain regions, meaning not all services will be or are (already) available in every region. So, make sure to check your region of choice on the availability of the service (s) you are interested in.
Concluding
Of course, there were many, many more releases (around Lambda, Groundstation, Deepracer, SageMaker, various IoT and IA services, machine learning, etc…) and announcements made during re:Ivent. Far too many for me to handle or discuss here. As such, I would encourage you to give the AWS re:Invent news blog page a visit. Jeff Barr, Chief Evangelist for AWS has done and is doing an amazing job getting all the information out there.
