Code Bug Fix: Returning an audio file as response from AWS Lambda function

Original Source Link

I have an AWS API Gateway calling a Lambda function (with Proxy integration), which should return an audio file.

Let’s say that I have my audio as ‘bytes’ object. How do I go about returning it, such that it can immediately be recognized and played by the browser (e.g. Firefox)?

Here’s what I got, but, clearly, that doesn’t really work.

sound: bytes

return {
    'statusCode': 200,
    'headers': {
        'Content-Type': 'audio/mpeg'
    'isBase64Encoded': True,
    'body': base64.b64encode(sound)

Solution for my case (proxy integration)

  1. Base-64 encode the bytes object (e.g. base64.b64encode(sound) in Python) and put the result in 'body' of your response dictionary.
  2. Set 'isBase64Encoded' to true in your response dictionary.
  3. In API settings, set Binary Media Types to */* and/or /.

Resources that helped me directly

Resources that might be useful

Tagged : / / /

Server Bug Fix: AWS CloudFormation creates new RDS instance

Original Source Link

I’ve created a CloudFormation script that stands up a clone of our existing AWS stack. I’ve added in information so that CloudFormation will create RDS from either a snapshot of an existing RDS instance, or create a new instance with a named database. However, when I try to apply this same script to our existing stack (verified by creating a new stack with the script from the existing stack and then attempting to upgrade via the new script), CloudFormation always creates a new RDS instance.

Extracted portions of the CloudFormation script are below.

    "Parameters": {
        "liveDbName" : {
            "Default": "default",
                "Description" : "The live database name to create by default",
                "Type": "String",
                "MinLength": "1",
                "MaxLength": "64",
                "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*",
                "ConstraintDescription" : "must begin with a letter and contain only alphanumeric characters."
        "liveDbSnapshotIdentifier" : {
            "Description" : "This overrides Default Production Db name",
            "Type": "String",
            "Default": ""
    "Metadata": {
        "AWS::CloudFormation::Interface": {
            "ParameterGroups": [
                    "Label": {
                        "default": "Db Layer Configuration"
                    "Parameters": [
    "Conditions": {
        "UseLiveDbSnapshot" : { "Fn::Not" : [{ "Fn::Equals" : [ {"Ref" : "liveDbSnapshotIdentifier"}, "" ] }] },
    "Resources": {
        "WebDb": {
            "Type": "AWS::RDS::DBInstance",
            "DeletionPolicy": "Snapshot",
            "Properties": {
                "AllocatedStorage": "100",
                "AutoMinorVersionUpgrade": "true",
                "BackupRetentionPeriod": "30",
                "CopyTagsToSnapshot": "true",
                "DBName" : {
                    "Fn::If" : [ "UseLiveDbSnapshot", { "Ref" : "AWS::NoValue"}, { "Ref" : "liveDbName" } ]
                "DBSnapshotIdentifier" : {
                    "Fn::If" : [ "UseLiveDbSnapshot", { "Ref" : "liveDbSnapshotIdentifier" }, { "Ref" : "AWS::NoValue"} ]
                "DBInstanceClass": {
                    "Ref": "webDbInstanceType"
                "DBParameterGroupName": {
                    "Ref": "WebDbParameterGroup"
                "DBSubnetGroupName": {
                    "Ref": "DbSubnetGroup"
                "Engine": "mysql",
                "MasterUsername": {
                    "Ref": "dbMasterUser"
                "MasterUserPassword": {
                    "Ref": "dbMasterPassword"
                "MultiAZ": "true",
                "PubliclyAccessible": "false",
                "StorageType": "gp2",
                "Tags": [
                        "Key": "Name",
                        "Value": "WebDb"

There are of course other portions of the script, but this is the portion (fully “namespaced,” I believe) that deals with our database section.

What am I doing wrong with my script, and is there a correct way to do so? Obviously I don’t want CloudFormation to restore a snapshot over our existing instance, but I don’t want it to create a new instance with the named database, either.

EDIT: “Existing” stack script included

I’ve added the existing stack script as a link to Dropbox because the file is too long to include here directly:

Your original CloudFormation template did not include DBName or DBSnapshotIdentifier properties. So the RDS instance was created without a DBName. Any database housed by your RDS instance was created after-the-fact, and not by CloudFormation.

Your new template includes either the DBName or DBSnapshotIdentifier, depending on the input parameters.

According to the CloudFormation reference docs for the AWS::RDS::DBInstance resource, if you add/change/remove either of the DBName or DBSnapshotIdentifier properties, then the RDS instance will be re-created.


I’m guessing that when you’re applying the updated template, you’re attempting to use the name of your database that you have inside your RDS instance as the value for liveDbName. However, as far as CloudFormation is concerned, this is a change to the RDS instance and requires replacement.

To apply the template update, you’ll need to modify it such that neither DBName nor DBSnapshotIdentifier are applied.

Tagged : /

Server Bug Fix: EC2 instance’s Docker ports not not accessible after changing instance type

Original Source Link

When changing the EC2 instance type I ran into a problem. The machine had 3 Docker containers that needed to be restarted and after the restart their ports became unreachable.

What could be the issue and how should I go about obtaining other necessary debug information?

  • No changes were made to the Security Groups configuration within AWS, all the required ports are still enabled.

  • I am still able to SSH into the EC2 instance, but the ports used by
    Docker (80, 8181) are not accessible (connection timeout).

  • Within a web browser it doesn’t matter if I’m trying to access a port that is being used or
    not, the browser behavior is always the same (loading indicator stops
    at the beginning, followed by timeout, nothing gets logged in e.g. Apache’s access.log or error.log

  • Within a web browser neither addressing the instance by its Public DNS (IPv4), IPv4 Public
    IP, or its original domain name works.

  • Restarting the instance or changing its type again doesn’t help

I am able to ping/telnet/wget the ports used by the Docker containers from
within the instance:

$ docker exec f227cf8d9481 wget
converted '' (ANSI_X3.4-1968) -> '' (UTF-8)
--2018-01-15 23:49:10--
Connecting to connected.
HTTP request sent, awaiting response... 401 Unauthorized

But not from the outside (the IP address still gets resolved):

$ wget <aws-ip>.<aws-zone>.<instance>
--2018-01-16 00:53:32--  http://<aws-ip>.<aws-zone>.<instance>
Resolving <aws-ip>.<aws-zone>.<instance>
Connecting to <aws-ip>.<aws-zone>.<instance>||:8181... failed: Operation timed out.

The docker containers are running and mapping between Docker ports is
done correctly:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                    NAMES
f227cf8d9481        cloud9              "forever /cloud9/s..."   3 seconds ago       Up 2 seconds>8080-8081/tcp, 80/tcp,>8181/tcp,>3000/tcp   my-cloud9
fa0d2bbce863        wordpress           "docker-entrypoint..."   59 minutes ago      Up 59 minutes>80/tcp,>443/tcp                                                 goofy_torvalds
6ada961a5ea0        mysql               "docker-entrypoint..."   About an hour ago   Up About an hour>3306/tcp  

Iptables setting seems to have the Docker ports enabled:

$ sudo iptables --table nat --list
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !loopback/8           ADDRTYPE match dst-type LOCAL

target     prot opt source               destination         
MASQUERADE  all  --  <aws-ip>.<aws-zone>.<instance>.internal/16  anywhere            
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:mysql
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:https
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:http
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:8181
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:tproxy
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.<instance>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:webcache
MASQUERADE  tcp  --  <aws-ip>.<aws-zone>.internal  <aws-ip>.<aws-zone>.<instance>.internal  tcp dpt:hbci

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
DNAT       tcp  --  anywhere             anywhere             tcp dpt:mysql to:<docker-ip>:3306
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:<docker-ip>:443
DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:<docker-ip>:80
DNAT       tcp  --  anywhere             anywhere             tcp dpt:8181 to:<docker-ip>:8181
DNAT       tcp  --  anywhere             anywhere             tcp dpt:tproxy to:<docker-ip>:8081
DNAT       tcp  --  anywhere             anywhere             tcp dpt:webcache to:<docker-ip>:8080
DNAT       tcp  --  anywhere             anywhere             tcp dpt:81 to:<docker-ip>:3000

No there doesn’t seem to be any noticeable network activity (around 2KB every 5 mins) using the Monitoring tool of the EC2 instance. Except for the some spikes from the times I used SSH to log in:

enter image description here

The issue was with the DNS and SSL certificate configuration, because the instance was configured to use HTTPS only..

After the change of the instance type, the new instance was automatically assigned a new URL, which had to be updated with both the DNS provider and the CA.

Tagged : / / / /

Server Bug Fix: ec2 instance connection timed out but can be pinged

Original Source Link

I’ve seen many similar questions but none like this.

So I started an ec2 instance and installed apache, WordPress, etc on top. And I was able to access everything fine with the public IP provided to me.

Then I created a hosted zone on route 53 to redirect my domain to this instance. This worked, but I wanted to create an elastic IP to associate with my instance.

As soon as it associated I can no longer access my instance.

I can ping, ssh to my instance no problem. However, when accessing the Elastic IP or public DNS using a browser, it just hangs there for seconds and shows connection timed out. Security rule allows all the HTTP https ssh ICMP etc.

I thought it could be the server’s problem, but when I ssh to instance it shows httpd is running fine.

I have deleted the hosted space, got a new elastic IP to associate, restart the instance, but at no point did the server become accessible. Short of making a new instance.

So I’m stuck here and would appreciate any help possible. If it helps this is the instance ID: i-09fcc30eb8776b7fe

Thank you

edit: attaching screenshot of my instance and my security group rules

for security group rules, when I select the source as “anywhere”, it auto-populates, and the next time I go in it becomes 2 rules.

With the advice from Tim, I ran curl -i address, finding that it shows 301 moved permanently.

Upon closer inspection, when I access the new public dns/elastic ip, it is still trying to connect to my old public ip which fails.

I created a new instance based on the image but when accessing the new instance public ip/elastic ip it still tries to connect to the old public ip.

In the end I created a new instance completely from scratch. This time I associate the elastic ip first, then redirect domain to the instance and everything worked.

(atm the browser still cant access my domain name because the ip is cached i think. opening in private/incognito and i see that it works. )

To allow an instance reachable on a particular port over the internet you have to make the following considerations.

  1. Make sure the instance is having a public IP address, this can be your elastic IP or a dynamic public IP provided by Amazon when you opt for it at the time of instance launch.
  2. The instance security group must allow the specific port from the internet where the source becomes – for IPv4 and ::/0 For IPv6 respectively
  3. If you are using NACL then you should also allow the specific inbound port and allow the ephemeral port range in the outbound. This is because NACL is stateless. Please check here for recommended NACL rules for your VPC

  4. Lastly you have to make sure the subnet in which this instance is launched has a route to the internet gateway (IGW). You may review the route table entries associated with the subnet.

Tagged : / / / /

Code Bug Fix: AWS Lambda Function Does Not Show Access to DynamoDb

Original Source Link

I am doing this AWS tutorial to create an AWS policy and role for a Lambda function to have access to a DynamoDb. After I add the roles to the Lambda function I should get a diagram like this one from the tutorial:

enter image description here

This is what I get:

enter image description here

Based on the comments.

The lambda function works as expected. Sometimes console will not display all the resources it has access to. This does not mean that it can’t access the required resources though.

Thus its better to check and see if it actually work despity of what console shows.

Tagged : / /

Server Bug Fix: When stopping EC2 instance, it starts again automatically listed separately, previous one changes to terminated

Original Source Link

I stopped our main instance to create and run a new one for temporary purpose. As I’m using free tier, I stopped the main one, created a new one, then in the instances, three instances appeared
The new one, the old main one which was earlier marked as stopped now terminated and a new main one marked as running. I stopped it again and again it appeared, now it’s showing me 4 instances.

enter image description here

This “Ejabberd” is the one I’ve created just now. The other ones appear on their own when I stop the running one. I’m new to AWS, I searched but couldn’t find anything about this behavior, I read how to stop an instance on their own website. What I want to do now is to stop it unless I want to start it and stop the “Ejabbered” one.

Edit – As suggested by user @ceejayoz about the auto scaling group, I checked it and found that there’s one enabled. I checked the official documentation to turn it off and found out that mine consists no scaling policy.

So, I looked more into the options of it and found out group details. On my own, I changed the desired capacity and Minimum capacity to 0 so it won’t bother when I stop my instance. It worked.

enter image description here

But, it changed my instance’s status to terminated from stopped. I don’t know how and why? Although, I recovered it back by re-changing the values of auto scaling group to original but why stopping an instance set it to terminated?

Your EC2 instance is managed by an Auto Scaling group.

EC2 instances that are managed by Auto Scaling may be launched and terminated at any time. It’s goal is to preserve a number of running instances greater than (or equal to) the Minimum count, and less than (or equal to) the Maximum count.. ideally the Desired count.

If you stop your EC2 instance, then Auto Scaling deems your EC2 instance as “unhealthy” and terminates it. It then replaces it with a new EC2 instance that it considers “healthy”. This restores the number of running instances back to the Desired count.

When you set your Auto Scaling group Desired count to 0, that told Auto Scaling that you didn’t want any running instances. It then terminated instances until the running count was 0.

When you reset the Desired count back to 1, you told Auto Scaling you wanted one running instance, so it launched a new one.

Auto Scaling did exactly what you it’s designed to do, and exactly what you told it to do.

Although there are exceptions to the rule, generally speaking:

  • if you want to manually stop and start your EC2 instances, don’t put them under Auto Scaling,
  • and, if you modify the contents of your EC2 instances directly, don’t put them under Auto Scaling.

I highly recommend reading up more about Auto Scaling, or don’t use it.

Tagged : /

Server Bug Fix: Shared files folder in Amazon Elastic Beanstalk environment

Original Source Link

I’m working on a Drupal application, which is planned to be hosted in Amazon Elastic Beanstalk environment. Basically, Elastic Beanstalk enables the application to scale automatically by starting additional web server instances based on predefined rules. The shared database is running on an Amazon RDS instance, which all instances can access properly. The problem is the shared files folder (sites/default/files).

We’re using git as SCM, and with it we’re able to deploy new versions by executing $ git aws.push. In the background Elastic Beanstalk automatically deletes ($ rm -rf) the current codebase from all servers running in the environment, and deploys the new version.

The plan was to use S3 (s3fs) for shared files in the staging environment, and NFS in the production environment. We’ve managed to set up the environment to the extent where the shared files folder is mounted after a reboot properly. But…

The Problem is that, in this setup, the deployment of new versions on running instances fail because $ rm -rf can’t remove the mounted directory, and as result, the entire environment goes down and we need restart the environment, which isn’t really an elegant solution.

Question #1 is that what would be the proper way to manage shared files in this kind of deployment? Are you running such an environment? How did you solve the problem?

By looking at Elastic Beanstalk Hostmanager code (Ruby) there seems be a way to hook our functionality (unmount if mounted in pre-deploy and mount in post-deploy) into Hostmanager (/opt/hostmanager/srv/lib/elasticbeanstalk/hostmanager/applications/phpapplication.rb) but the scripts defined in the file (i.e. /tmp/ don’t seem to be working. That might be because our Ruby skills are non-existent.

Question #2 is that did you manage to hook your functionality in Hostmanager in a portable way (i.e. by not changing the core Hostmanager files)?

Have you considered using sources via .ebextensions with your shared files stored in S3?

I have used sources to deploy shared files (sitemaps) that are built by our app server (and stored in S3).

  "c:/inetpub/sitemaps": http://[S3 bucket]/

    command: copy c:\inetpub\sitemaps\*.* c:\inetpub\wwwroot\ > copy_sitemaps_to_www.log 2>&1
    waitAfterCompletion: 5

S3fs is a bad idea, performance-wise, and also S3 is not designed to be mounted, it’s not a filesystem, it’s an object storage.

Use EFS instead ->

You will see in the documentation that there is a procedure for Drupal in Beanstalk.


Tagged : /

Code Bug Fix: Moving data from RDS to S3 using Glue

Original Source Link

I have a table in Amazon Arora Postgres. I need to move that table to S3 bucket in csv format. I have create the following pyspark code in AWS glue. Instead of storing as a csv file in S3 bucket. Multiple files are created in S3 bucket like run-XXX-part1. Is there a way to export a rds table into csv file in S3.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "test1", table_name = "testdb_public_reports3", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
## @type: ApplyMapping
## @args: [mapping = [("orderapprovedby", "string", "orderapprovedby", "string"), ("lname", "string", "lname", "string"), ("unitofmeasurement", "string", "unitofmeasurement", "string"), ("orderrequesteddtm", "timestamp", "orderrequesteddtm", "timestamp"), ("orderdeliverydtm", "timestamp", "orderdeliverydtm", "timestamp"), ("allowedqty", "decimal(10,2)", "allowedqty", "decimal(10,2)"), ("addressid", "int", "addressid", "int"), ("fname", "string", "fname", "string")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("mname", "string", "mname", "string"), ("lname", "string", "lname", "string"), ("designation", "string", "designation", "string"), ("joiningtime", "timestamp", "joiningtime", "timestamp"), ("leavingtime", "timestamp", "orderdeliverydtm", "leavingtime"),("fname", "string", "fname", "string")], transformation_ctx = "applymapping1")
## @type: DataSink
## @args: [connection_type = "s3", connection_options = {"path": "s3://deloitte-homefront-poc/PROCESSED"}, format = "csv", transformation_ctx = "datasink2"]
## @return: datasink2
## @inputs: [frame = applymapping1]
datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://path"}, format = "csv", transformation_ctx = "datasink2")

Using glue and pyspark just for exporting data is not a good option. You can follow the step by step guide provided by aws

Of you still want to use Glue and want single output file

datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://path"}, format = "csv", transformation_ctx = "datasink2")


Tagged : / / / /

Server Bug Fix: Elastic Beanstalk S3 access via .ebextensions

Original Source Link

I have a simple file in my .ebextensions folder:


                    type: S3
                    roleName: aws-elasticbeanstalk-ec2-role
                    buckets: my-bucket
        mode: "000755"
        owner: root
        group: root
        authentication: S3Access

Which according to multiple answers is the way to grant S3 bucket access to the aws-elasticbeanstalk-ec2-role role.

But I continue to get the 403 error in /var/log/eb-activity.log

[2015-08-26T01:27:03.544Z] INFO  [22320] - [Application update/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild] : Activity execution failed, because: Failed to retrieve HTTP Error 403 : <?xml version="1.0" encoding="UTF-8"?> (ElasticBeanstalk::ExternalInvocationError)

If I manually add an S3 access policy to the aws-elasticbeanstalk-ec2-role role everything works, so I know I don’t have misspellings in URLS or anything else, the EC2 instance is definitely in the correct role.

What is wrong?

PS. I tried the files section with or without the ‘authentication’ setting.

I’ve figured it out and I feel a little bit silly for not picking this up sooner.

So for anyone that uses AWS::CloudFormation::Authentication path, the solution of course is:

Make sure your BUCKET policy allows your aws-elasticbeanstalk-ec2-role. DOH!!

It should look something like this:

    "Id": "Policy1111Blah",
    "Version": "2012-10-17",
    "Statement": [
            "Sid": "Stmt1440Blah",
            "Action": [
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::my-bucket/*",
            "Principal": {
                "AWS": [

You can grab the ARN from IAM console.

The instructions in your .ebextensions config files only tell the EB deploy tools what to use to authenticate, but your source bucket (if private obviously) needs to allow that principal access!!!

Tagged : / / / /

Code Bug Fix: I want to be able to download a directory on S3 as a zipped file

Original Source Link

So basically I have a directory on S3 lets say dir/ which contains a bunch of images and videos. I want to be able to download the whole directory as a file. What would be the best way to do this?

Would prefer a language agnostic solution. If not I am working with Python.

Tagged : / /