Researchers discovered incompetence when handling AWS RDS database backups, exposing them on the public network for anyone to take
File this under “Disheartening that we are still seeing this“. AWS has this feature where you can make your RDS backups public, making it easy to share them with other AWS accounts to spin up new database instances. Doesn’t mean you should though.
Research team over at MITIGA have published a very exhaustive look at their analysis on how they discovered not only a huge amount of publicly available data backups, but of those, an alarmingly large amount of them that had Personally Identifiable Information (PII) data.
We wouldn’t even call this hacking. Incompetent cloud engineers have packaged up the complete database, and left it out on the doorstep for anyone, passing by, who is interested in it. No need to worry about breaching firewalls, network layers, or even guessing at username/password combinations.
The article’s statistics are demoralizing to anyone in the cloud and security space.
The total number of snapshots seen in this month (Oct 22) was 2,783
Of those 2,783 snapshots, 810 snapshots were exposed during the analyzed timeframe
Additionally, 1,859 snapshots of the 2,783 were exposed for just 1-to-2 days
There simply is no excuse for this sloppy and incompetent practice. It demonstrates a complete lack of respect for security and instead of going down the path of securing with IAM roles, they just thought it easier to make it public. Ease of use I am sure, they said to themselves, convincing themselves no one will know.
As the old saying, with great power comes great responsibility. Learn the tools. We need more name and shaming so people start taking this more serious. Don’t just name the company, but the head of security or cloud that allowed this to happen under their watch.
What makes a great product by AWS guru James Hamilton
James Hamilton, a true computing legend who has architected many revolutionary things over at AWS that us mere mortals can only dream and be in awe of. I have followed him for many years, and have always taken the attitude – when James talks, you listen.
I have come across many product managers in our portfolios over the years, some good, some poor, and some that shouldn’t be anywhere near the product. This is a hard role to get right or even define properly, but you know it, when you see it done right.
James has a great take on what it takes to be a great product manager especially inside of AWS. I would caution though, that I am going to assume he is referring to the great managers that design their core AWS products, and not the ones in charge of the AWS Console (it is getting better, but it still feels like an after thought, a reluctant layer put on top of what is already a great API).
If you ever find yourself with some time on your hands, and would like to get a peak under the covers at AWS, then do a YouTube search for the sessions that James has given at re:Invent over the years. Mind blowing some of the things they have done to get the performance, scale and uptime out of AWS that we all rely on.
Amazon has closed the gap between Lambda and processing events directly from SQS, opening up a world of possibilities.
This week Amazon announced that they finally closed the gap between it’s oldest service, Simple Queue Service (SQS) and their push into serverless computing, Lambda. In other words, you can now write Lambda functions (Java, JavaScript, Go, Python, C#), point them at an SQS queue and have them processing events off the queue all without worrying about supporting a complex server environment.
Anyone in the cloud architecture or microservices business will be excited at this long overdue evolution of the Lambda ecosystem. When Lambda was first introduced, the idea of having functions hanging out there in the cloud, just waiting to be executing based on some sort of condition was very exciting.
An aside
As any seasoned architect will tell you, one of the secret tools of a good scalable architecture is liberal but strategic use of queues.
A queue allows you to decouple components from one another to the point of where downstream execution can be many magnitudes later, making the overall resilience of the platform much stronger at the same time as far more scalable. For the sake of a quick illustration, imagine two components that are tied together. One has to process an order, and the other is responsible for emailing out the confirmation. You may split these up into two different services and when the order has been completed, you could make a RESTlet call to the service to send out the email. But what happens if that email service is no longer there, or returns an error? Now you have to start developing retry logic and figure out what state you want to leave that order with respect to the customer.
Instead of a strong coupling like this, the better solution would be a queue between both components, where the ordering service would place a message on the queue that the email service would pick up at a later date to process, resulting in an email going out. If the email service is down, then no problems, the queue can still accept events and will store them until such times the email service can process them.
dreaming
When I first started dreaming of using Lambda it was for queue processing. There was many times that the amount of business logic required to process queue’s would have fitted wonderfully well within the Lambda environment, all without having to worry about standing up containers or EC2 instances. However, I had to keep dreaming because this was not available at the time of Lambda launch – one of the most obvious use-cases of Lambda and Amazon made us wait nearly 4 years for it.
The wait is over and now we can start using Lambda for some serious queue related applications. Yes, you can put some quite meaty processing behind each event on a queue and that will make architectures much easier to manage and scale, but the real power behind this development is actually in making some quite sophisticated event routing applications.
Imagine taking a source event from a given queue and then deciding, based on its contents, which additional queues it should be placed on for parallel processing. Take the example of the order system in the sidebar; the email service is not the only service that should know about an order being complete, there could many others in the enterprise that could benefit from that information, for example fulfillment, accounting, warehouse to name a few. A Lambda function could take that order from the queue, and decide quickly which other queues should have a copy of that event.
Such routing design patterns are common and historically you’ve either had to use some non-cloud technology to perform these, or grown your own. This Lambda tie-up simply reduces the amount of infrastructure required to support such a design.
but wait
While the wait is over, Amazon has not made it as clean as I would have hoped. There is a little sting in the tail and it gives a little clue as to how they are providing this service under the covers.
One of the shortcomings (though I understand why) of SQS is that it requires the client to do a long-poll to determine if there is events on the queue to be processed. In other words, you had to keep making an HTTPS call “do you have any events for me?” and while the call would wait for a period of time before returning back, you had to do this all the time. Each call to SQS, yes, you were getting charged for it.
For large systems that have a constant stream of messages coming through, this cost is negligible, but for systems with sporadic bursts, this overhead could be costly and above else, in-efficient. Traditional messaging systems, would keep a constant connection to the messaging service and events would be delivered down the wire instantly.
To minimize this overhead, it is not uncommon to have one consumer of a queue who would do the polling and then pass out the work to a battery of internal threads. This is an environment where you would have a server handling queue event processing that was capable of executing multiple threads at once.
The Lambda world however, is serverless, so you have to forget about the underlying platform of real servers that Amazon manages for you to give you this illusion of serverless computing. Since they charge only for the function execution, as far as you are concerned you are allowed to forget the underlying server.
SQS has not changed its behavior. You still have to poll to retrieve messages and this is what Lambda is doing under the covers for you. However this time, you have no real control over the amount of consumers that will be running up that SQS bill for you, particularly if you have come off of a very large volume of events that Lambda scaled out to execute in parallel all those events.
No doubt Amazon have thought of this and will be monitoring the situation and taking the necessary steps to reduce this overhead. In an ideal world, they would figure out a way for the Lambda service to have a continuous direct connection (think websocket for queues) to the SQS service so when an event came in, they could instantaneously hand it over to a Lambda function for execution.
in the meantime
Until that time, this is a huge step forward and makes the use of serverless computing even more attractive as design out the next generation of cloud solutions.
Moving from RackSpace to FastMail as I trial a new email service provider with more features and security.
I have had my private email handled by RackSpace for many years now. I never really had a major problem with RackSpace, offering quality email hosting at a very low cost ($2 per mailbox per month), and a nice web interface as well as secure IMAP access.
I utilize wildcard email addresses – (*@mydomain goes to a single mail box) when I sign up for services. This way I can track who sells my data (and some of the big boys have been found out over the years) and more importantly shutdown the email if it no longer serves the purpose.
It is with this last requirement that RackSpace struggled with. Something changed, somewhere, but their incoming rules and spam filters weren’t really doing the job any longer. Maybe it was time to have a look around.
I had three main requirements from my email host (outside of the usual security)
Support secure IMAP
Provide wildcard/catch-all email address
Provide alias support that can email external email addresses
I have been keeping my eyes open as various email providers pop up, especially in the light of the whole Edward Snowden, claiming to be super secure. However, even they lacked the ability to support my 3 requirements. Enter FastMail.
FastMail was previously owned by Opera (the browser people) and then it was spun out as a separate entity, based down under in Australia. They are a subscription only service, which basically means, no free-loaders and therefore earning enough money to stay in business without relying on advertising.
I am currently trialing their service for free for 30 days, then it will be $50 a year (for 25GB). This is definitely more expensive than RackSpace ($24 per year for 10GB) but still around the ball park for email services. In addition, FastMail can also host your domain with that price. They do have a $30 per-year 10GB service too, but no custom domain.
With respect to security, they are outwith the reach of the US government for any inspection of your inbox as well as pre-loading any remote images on their servers so you won’t get dinged with unnecessary tracking or snooping by remote loaders. Speaking of security, the standard username/password gets you into the website, but you must create separate passwords for any application wishing to access your inbox via IMAP. I really like this extra touch – including all the tracking of who/what has been accessing your inbox.
I am actually going to try and live with their web-ui and stop using Thunderbird as my email client. I like their UI, extremely fast and responsive. It interacts with the desktop beautifully so we will see. They have a dedicate mobile app that is very sweet. So no complaints on my tablet or phone.
But what really swayed it for me, was their rule filtering. They give you access to the underlying Sieve scripts. This makes it extremely powerful and flexible to do whatever crazy ass rule concoctions that I can dream up.
Couple of other nice features are popping up as I use it more, including 2-way synchronization with Google Calendar, custom login screen and extremely easy to import data from another IMAP provider.
I will see how it goes. I am not deleting my RackSpace account just yet, but gut tells me I will be sticking with FastMail.
Amazon WorkSpaces is basically a remote Windows 7 desktop with a couple of billing options. Discover which works best.
Amazon WorkSpaces, put basically, is a Windows 7 desktop running in the Amazon cloud. You treat it like any other Windows machine, except instead of using it on your local physical machine, you remote desktop into it from any device (including mobile or a tablet) and use it remotely. Amazon manages all the hardware and underlying operating system for you (sadly only Windows 7 is available at present).
Extremely handy for people that are on the move, security sensitive applications, or need to share a desktop (not simultaneously) with colleagues. You can setup all the software you need, snapshot the installation and spin up as many instances as desired.
Amazon provides a secure, hassle-free, client to which to access these desktops in the cloud. Not unlike Windows Remote Desktop, this client gives you access to the remote full desktop, including mapping local printers and sound devices. You don’t however get the ability to transfer files up and down or access USB drives – making it particularly useful for applications that require a certain level of security.
There is a lot of management tools that come with WorkSpaces that makes it powerful for Administrators to provision and control the machine so the users have minimal access to the core machine. For example, when a user boots up, they get their own local D: drive that is persisted between reboots. The traditional C: drive is hidden from view and will be reset upon restarts. An administrator can install software in the traditional way and then create an image from that so other machines may be spun up with that configuration.
There are 3 different flavors available at present:
Value : 1 vCPU, 2 GiB Memory, 10 GB User Storage ($25 per month)
Standard: 2 vCPU, 4 GiB Memory, 50 GB User Storage ($35 per month)
Performance: 2 vCPU, 7.5 GiB Memory, 100 GB User Storage ($60 per month)
Incidentally if you bring your own Windows license, you can save $4 per month from each of those prices. You can also add in an application bundle (MS Office Professional mostly) for an extra $15 per month.
You will notice though that this is not the typical per-hour Amazon billing. Amazon released this service with a fixed monthly pricing model. This was a little disappointing. No matter if you run up the service for 1 hour or a full month; you were paying the same. What is this? Rackspace?!? This is not the Amazon way.
Amazon listened and just last month they released a new pricing option that lets you pay for what you use. The billing stops when you log-off and does not get started until your log back in again. However before you get too excited, there is the question of where does the data go when you are logged off? Who pays for that?
You do.
If you opt for hourly billing, you have to pay a small monthly surcharge with each WorkSpace you spin up. For example, per month the Value package will cost you $25, or alternatively you can pay the $7.50 surcharge and then pay $0.22 per hour you are logged in for.
So what works out cheaper? Monthly or on-demand? Well it all comes down to just how much you are intending to use the machine. For example, if you have your accounts software on your machine, then chances are you won’t be on the machine continually – maybe only a few days of the month.
Making some assumptions about the average working month, we can give ourselves a rule of thumb of when you should use one over the other. The assumptions made here are that there is no weekend work and the average month has 20 working days.
If you do that and crank the numbers, you discover, that the break-even point is around the 4 hrs per day mark. Anything more than that, then you are giving Amazon more money than they deserve.
On the face of it, 4hrs per day doesn’t sound a lot. However, for the situations we are using Amazon WorkSpaces the per-hour billing will save us considerably. It has to be noted, that when you are logged off, the machine is essentially in a sleep state – that means any backup or maintenance software you want to run up in the middle of the night won’t. Conversely, a secure Windows machine is one that is turned off. While the security of Amazon can be setup to be very secure, nothing beats turning off a machine to maintain ultimate deniability to would-be hackers.
WorkSpaces has proven to be really useful for us at ParkerGale, particularly as we manage many different portfolio companies and the variety of VPN/firewalls hoops each company throws at us. I will be evaluating WorkSpaces for my own private development machine over the next few months to see how well it stands up to having everything on my own laptop.
The ease, the security, the flexibility and more importantly the cost, make WorkSpaces a very compelling offering for a company that was about to purchase laptops/desktops for their office. That cost can be offset and instead issued very dumb down Chrome books. That CapEx is once more turned into OpEx without the depreciation or upkeep (or headache upon failure) of the hardware.
So before you go purchasing, run the numbers, take it for a spin and see how you get on.
How can you be confident your platform is scalable? As a CEO what questions do you need to ask your engineering team.
How often have you heard people throw around the scalable word? Often used in a boastful manner to illustrate they are ready for scale. Being a technologist I treat such boasts with much skepticism; because scaling an infrastructure or enterprise is bloody difficult and near on impossible if not been properly designed from the outset.
Every deal book we see at ParkerGale they all describe their systems as a “scalable platform” – but rarely they are. Too much faith has been put in the wonders of the cloud, which many read and assume that their scaling problems are magically over.
In this article I am going to help the CEO to determine if their platform is ready for the scale they have been told it is.
Because the cloud makes it quick, easy and cheap to spin up servers/resources the assumption is made that scaling is linear. Let’s look at the root of the problem with a simple example and maths that makes it look easy on paper.
Your web site is running on a single server and has never shown any sort of strain. Every one of your 1000 customers get a fast response and never been denied their experience. Great job engineering team.
“No problem” yells the CEO, “add another server“
The business decides to double its clients – 2000 customers hitting on your site. “No problem” yells the CEO, “add another server“. Since 1 server handled 1000 customers, 2 servers can handle 2000 clients. Problem solved.
This logic is hard to argue with on face value. However the devil is in the detail as you recall the classic phrase about how 9 mothers can’t make a baby in 1 month no matter how dedicated each mother is to the process.
While many systems can have more “mothers” thrown at it, a lot of the core systems can’t scale like that. The big one that most struggle with is the good old database; the heart that pumps the data around the company. You generally want all your data in one place so it maybe an analyzed and queried. This comes at a huge cost. Often we see lots of front-end application servers all properly load balanced and designed, only to be failed by the single database serving them all.
Application servers can prove troublesome too; nothing worse than having your session data loaded on one server only to have the another server deal with your request because the load balancer decided to balance you off to another. Not insurmountable, but you have to design for this from the start.
There is an assumption that because a company is utilizing the infrastructure of a cloud provider they have outsourced the scaling problem. If I have 100 sales people doesn’t mean I am going to have x100 the amount of sales. They still need managed and properly allocated their workload. All a cloud provider does for a company is to provision resources quickly.
You can validate your scalability readiness by asking your engineering team some questions.
What steps did they take during development to provision for scale?
Teams will often outsource this to the underlying platform/framework they chose. For example if say that .NET/JEE takes care of it automatically; #RedFlag
What is the weakest point of the system?
No matter how great the system is, there is always at least one weakness. If they can’t identify any; #RedFlag
What level of scale can the system cope with?
Any type of scale! #RedFlag While you don’t need the scale of Facebook, they even have scaling woes.
How much testing was done to confirm the scaling touch points?
If they look at you, head tilted like a dog trying to understand what its master just said; #RedFlag The great follow-up question here is to ask at what point did the system break? If they couldn’t get it to break then the testing was not thorough enough.
What about down scaling?
This is a great question and usually puts a team into a bit of a panic. Bit like asking if anyone has tried to restore any of the backups. If they have never done this; #RedFlag Scaling back down, turning off resources and making sure you lose nothing, is actually as hard, if not harder, than continually adding to a system.
Scaling an infrastructure is a very difficult task, even if you have the luxury of green-field developing it from scratch, but it is 10x harder if you are trying to make a legacy system scale.
Keep some perspective and do not assume the problem to cope with demand is someone elses – cloud platforms will only take you so far. Development and consultancy shops talk a great game but they too can fail the 5 Quick Scale Questions.
No matter how convincing someone is, scaling an enterprise is surprisingly difficult.
alan williamson 