How to create API Gateway deployments?

[Mathias Sunardi]

I'm trying to use LocalStack for locally testing a serverless application. It needs some API Gateway APIs to trigger some Lambdas. I could not create the API deployment. I did the following:

from localstack.utils.aws import aws_stack

api = aws_stack.connect_to_service('apigateway', env=None, client=True)
response = api.create_rest_api(name='TestApi',  description='This is a test API',
    version='0.1')
# response:
# {'ResponseMetadata': {'HTTPHeaders': {'date': 'Thu, 30 Mar 2017 17:57:34 GMT',
#   'server': 'BaseHTTP/0.3 Python/2.7.12'},
#  'HTTPStatusCode': 200,
#  'RetryAttempts': 0},
# u'createdDate': datetime.datetime(2017, 3, 30, 17, 57, 34, 799000, tzinfo=tzutc()),
# u'description': u'This is a test API',
# u'id': u'3795578091',
# u'name': u'TestApi'}
api.get_rest_apis()
# Returns:
# {'ResponseMetadata': {'HTTPHeaders': {'date': 'Thu, 30 Mar 2017 18:10:43 GMT',
#   'server': 'BaseHTTP/0.3 Python/2.7.12'},
#  'HTTPStatusCode': 200,
#  'RetryAttempts': 0},
# u'items': [{u'createdDate': datetime.datetime(2017, 3, 30, 17, 57, 34, 799000, 
# tzinfo=tzutc()),
#   u'description': u'This is a test API',
#   u'id': u'3795578091',
#   u'name': u'TestApi'}]}

api.create_deployment(restApiId='3795578091', stageName='v1')
# Returns:
# {'ResponseMetadata': {'HTTPHeaders': {'date': 'Thu, 30 Mar 2017 18:16:11 GMT',
#   'server': 'BaseHTTP/0.3 Python/2.7.12'},
#  'HTTPStatusCode': 200,
#  'RetryAttempts': 0}}

api.get_deployments(restApiId='3795578091')
# Returns:
# {'ResponseMetadata': {'HTTPHeaders': {'date': 'Thu, 30 Mar 2017 18:16:18 GMT',
#   'server': 'BaseHTTP/0.3 Python/2.7.12'},
#  'HTTPStatusCode': 200,
#  'RetryAttempts': 0},
# u'items': []} # << nothing?

I also tried adding more resources/methods, and using the CLI with the same result. I'm hoping to create some APIs using Swagger files so I can match the actual APIs. I wanted to test my application by sending a POST payload through the API URIs, which then go through a whole workflow processed by Lambdas, SQS, DynamoDB, etc ... and checking that the end result is correct.

Without the API deployments I am not able to get the URIs. Am I not using LocalStack correctly?

[Peter McGuire]

Same here. Using the CLI:

aws --endpoint-url=http://localhost:4567 apigateway create-deployment --rest-api-id 025A-Z746A-Z0A-Z --stage-name prod

Returns nothing and no errors.

Then,

aws --endpoint-url=http://localhost:4567 apigateway get-deployments --rest-api-id 025A-Z746A-Z0A-Z

Returns:

{ "item": [] }

[Waldemar Hummer]

Thanks for reporting @msunardi and @peterecofit . This bug is fixed in commit https://bitbucket.org/atlassian/localstack/commits/ce2d37e227c1791f3c8d5dd7ccd26becba379f82 . Can you please pull the latest version (from git, or the latest Docker image) and give it a try. Thanks

[Peter McGuire]

Hi @w_hummer, I can confirm that I am seeing deployments being created as well as associated stages.

However, I'm not seeing anything from generated URI:

aws --endpoint-url=http://localhost:4567 apigateway get-stages --rest-api-id 6407637205

Returns:

{ "item": [ { "description": "", "stageName": "prod", "variables": {}, "cacheClusterEnabled": false, "deploymentId": "A-Z988987047", "methodSettings": {} } ] }

Good!

However, using:

INBOUND_GATEWAY_URL_PATTERN.format(api_id='6407637205', stage_name=prod, path='/')

Returns:

http://localhost:4567/restapis/6407637205/prod/_user_request_/

And this URI results in a 404. Hitting just http://localhost:4567/restapis/6407637205 does actually return expected result:

{"description": null, "createdDate": "2017-05-14T17:33:24.616Z", "id": "6407637205", "name": "beacontest"}

[Waldemar Hummer]

@peterecofit This is fixed in commit https://bitbucket.org/atlassian/localstack/commits/3403f6d731b4bbc6ae2d9b8dd4625c2985183998

See here for an end-to-end use case with API Gateway and HTTP backend: https://bitbucket.org/atlassian/localstack/src/master/tests/integration/test_api_gateway.py?at=master&fileviewer=file-view-default#test_api_gateway.py-98

[Mathias Sunardi]

Hi Waldemar. Thanks for the response.

I am still unable to get a deploymentId. See attached screenshot.

And for the test_api_gateway.py script I'm getting a 404

I have the localstack container running using

make docker-run

I'm assuming it also rebuilds any changes that needs to be rebuilt. Any ideas on what I could be doing wrong?

FYI, my localstack repo is at commit 64ca6e1518e1bd3acc7d24d208c02fa325b8e190. In git log, that commit is after the 3403f6d commit, and shows as merge of commit 85c325d and 3403f6d. Do I need to rollback to 3403f6d?

[Waldemar Hummer]

I'm assuming you are using the latest version of the Docker image, but can you make sure and run the following commands manually (This currently does not happen automatically when you run make docker-run, we should probably add this).

docker pull atlassianlabs/localstack
docker pull atlassianlabs/localstack:latest

It still seems to me that you may have to create the REST endpoints and methods/integrations on your API Gateway, though. It actually requires a series of API calls to get a fully functioning API Gateway up and running. If you look at the integration test ... https://bitbucket.org/atlassian/localstack/src/master/tests/integration/test_api_gateway.py?at=master&fileviewer=file-view-default#test_api_gateway.py-72

... the code calls aws_stack.create_api_gateway(...) to create the gateway and REST resources. An integral part of this is to call create_resource(...) (to create the REST resource under a given path, say /hello_world) and then to call put_method(...) to add methods to that endpoint ...
https://bitbucket.org/atlassian/localstack/src/master/localstack/utils/aws/aws_stack.py?at=master&fileviewer=file-view-default#aws_stack.py-334

... and finally calling put_integration(...) to define the behavior of the API endpoint methods (e.g., forward requests to a Kinesis stream, or forward requests to another HTTP endpoint, etc.)

I'm not entirely sure why the test fails for you, but I haven't tried running it directly (I usually run make test to kick off the entire test suite).

[Mathias Sunardi]

Thank you, Waldemar! I will try your advice first thing in the morning.

[Mathias Sunardi]

I did the docker pull and that indeed helped. I think I'm at the same point as Peter McGuire now where I get a 200 on the root, but 404 on a resource. Attached is my result.

I tried the following urls, and both got 404

'http://localhost:4567/restapis/9438850502/FooResource'
'http://localhost:4567/restapis/9438850502/v2/FooResource'

[yezhongkai]

I got same doubt, i create Rest Api, Resource, Method, Integration, Integration Response and Method Response and Deployment Here are some screenshot of my test

But I don't know how to test that, what the rule for the deployment url?

I try the

http://192.168.3.44:4567/PatentApi

but get 404~

[Mathias Sunardi]

@yezhongkai The url for your api is in the format: http://<ip or localhost>:4567/restapis/<restApiId> So in your case the base path for your API is: http://192.168.3.44:4567/restapis/1892A-Z90081 By default, the method is GET.

However, I'm still getting 404 for any resources I added regardless of method (e.g. POST/GET) ... even after deploying with a new stagename.

[Mathias Sunardi]

Update: adding /deployments/<deploymentId> at the end of the path also gives a 200.

E.g. requests.get('http://localhost:4567/restapis/09A-Z4404753/deployments/6A-Z49917410')

But I still get a 404 when adding my resource path to it, even after creating a new deployment.

[Waldemar Hummer]

The problem is that API Gateway uses a generated domain name for deployed APIs , see docs here: http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-call-api.html

https://{restapi_id}.execute-api.{region}.amazonaws.com/{stage_name}/

Since custom domain names are not available for local testing, in LocalStack we use the convention of adding a hardcoded _user_request_ to the URL. The URL scheme for invoking APIs is as follows: (see here https://bitbucket.org/atlassian/localstack/src/master/localstack/config.py#config.py-67)

'http://localhost:4567/restapis/{api_id}/{stage_name}/_user_request_{path}'

For example:

'http://localhost:4567/restapis/09A-Z4404753/v1/_user_request_/mypath'

@yezhongkai @msunardi Please let me know if that works for you (I will update the documentation to make this more explicit). Thanks

[yezhongkai]

Hi @w_hummer i download the latest docker image and recreate all the resources, the status is like below:

And also, i deploy it to a stage:

And i try it as below:

Then u can see, i got an error, but i am very sure the Api that been registered in the integration can give response as below:

I check all the operation and can't find anything wrong, can u give a full example of Api Gateway?

[Waldemar Hummer]

The API definition looks good to me, but we have another problem now. As you are running LocalStack in a Docker container, the API Gateway service attempts to connect to the target service from within the container, hence you need to ensure that the service is actually accessible (in terms of network). I would assume that this is not the case.

I basically see two possible options:

1) running LocalStack locally (make infra) on the machine host instead of inside Docker. That allows API Gateway to access all services on localhost (and 192.168.3.248 in your case)

2) Instead of 192.168.3.248, using the IP address of the docker bridge (usually 172.17.0.1) in the target URL of your API Gateway resource. Note that this works only in Linux and will not work in MacOS (or Windows), because of the double indirection which essentially isolates the sub-networks (MacOS Host -> Linux VM -> Docker container).

[yezhongkai]

My test server is Linux(The server firewalld has been closed, in order to let docker could connect to each other), i go into the Localstack docker and use curl command sent request to 192.168.3.248. The network have no problem, can get the response

So i think my case is not related to the network issue~

[Waldemar Hummer]

OK that's interesting, something else must be the problem then. Can you please run LocalStack with debug output enabled:

DEBUG=1 make docker-run

And then please post the relevant output here (ideally the error stack trace, if there is any). Meanwhile I'll also try to reconstruct the example locally for testing. Thanks

[Waldemar Hummer]

Unfortunately I still cannot reproduce the issue. If you take a look at the example below, can you confirm that these commands work for you?

$ api_id=$(aws --endpoint-url=http://localhost:4567 apigateway create-rest-api --name api1 | jq -r .id)
$ root_res_id=$(aws --endpoint-url=http://localhost:4567 apigateway get-resources --rest-api-id $api_id | jq -r .items[0].id)
$ new_res_id=$(aws --endpoint-url=http://localhost:4567 apigateway create-resource --rest-api-id $api_id --parent-id $root_res_id --path-part mypath | jq -r .id)
$ aws --endpoint-url=http://localhost:4567 apigateway put-method --rest-api-id $api_id --resource-id $new_res_id --http-method GET --authorization-type NONE
# connect the API method to the S3 service (http://localhost:4572/) for testing
$ aws --endpoint-url=http://localhost:4567 apigateway put-integration --rest-api-id $api_id --resource-id $new_res_id --http-method GET --type HTTP --uri http://localhost:4572/
$ curl http://localhost:4567/restapis/$api_id/v1/_user_request_/mypath
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01">
  <Owner>
    <ID>bcaf1ffd86f41161ca5fb16fd081034f</ID>
    <DisplayName>webfile</DisplayName>
  </Owner>
  <Buckets>
 </Buckets>
</ListAllMyBucketsResult>

I think it will be essential for you to provide a reproducible test case. Can you please provide a self-contained script with either 1) the aws CLI commands as above, or 2) the Python commands to create the API endpoint and invocation that leads to the error. Sorry that this is a bit painful to debug, but you may have encountered a weird edge case or something. Thanks

[yezhongkai]

@w_hummer Thanks for ur patient, i will have a debug recently and let u know~ :)

[Former user]

Hi, I'm trying to tie lambda functions to an API Gateway too and might be running into something similar. To test this out, I've downloaded the latest localstack docker image (62cf31df11f1 created 2 days ago) and walked through the AWS tutorial on the subject here: http://docs.aws.amazon.com/lambda/latest/dg/with-on-demand-https-example.html

If you'd like to see the issue I'm running into, you can try it out by using the gist I've made here: https://gist.github.com/artburkart/508424db27b886a14fea577b13a73ed8 The only gotcha with the bash script in that gist is that it depens on jq, so you may need to install it before the test case works.

Essentially, it doesn't seem like the apigateway test-invoke-method functionality is working properly. I'm going to double back and make sure the code I've written works properly in AWS and check whether the localstack _user_request_ convention you documented here works for me or not. More soon.

Thanks for the awesome project!

[yezhongkai]

Hi @w_hummer i add the DEBUG=1 to my environment file. but i can't find any log output directory. Could u tell me where can i check the log? :-)

And i also try to use CLI to follow your step to reproduce the result, i am sure my CLI is ok, but i got:

$ aws --endpoint-url=http://192.168.3.44:4567 apigateway create-rest-api --name PatentApi

An error occurred (500) when calling the CreateRestApi operation (reached max retries: 4): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

So i paste my example here first, the test case is running on REST Api, here are the steps:

1. POST restapi:create http://192.168.3.44:4567/restapis

{
  "name" : "PatentApi",
  "description" : "PatentApi",
  "version" : "V2.0.0"
}

2.GET restapi:resources http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources

3.POST resource:create http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources/8656935207

{
  "pathPart" : "title"
}

4.PUT method:put http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources/8656935207/methods/GET

{
  "authorizationType" : "NONE",
  "apiKeyRequired" : "false",
  "operationName" : "GetTitle"
}

5.PUT integration:put http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources/8656935207/methods/GET/integration

{
  "type" : "HTTP",
  "httpMethod" : "GET",
  "uri" : "http://192.168.3.248:9910/openapi/2.0.0/monitor"
}

6. PUT integrationresponse:put http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources/8656935207/methods/GET/integration/responses/200

{
  "selectionPattern" : "2\\d{2}",
  "responseParameters" : {
    "method.response.header.Content-Type" : "'application/json'"
  }
}

7.PUT methodresponse:put http://192.168.3.44:4567/restapis/A-Z63A-Z075688/resources/8656935207/methods/GET/responses/200

{
  "responseParameters" : {
    "method.response.header.Content-Type" : "false"
  }
}

8.POST stage:create http://192.168.3.44:4567/restapis/A-Z63A-Z075688/stages

{
  "stageName" : "PatentApi",
  "deploymentId" : "String",
  "description" : "String",
  "cacheClusterEnabled" : "Boolean",
  "cacheClusterSize" : "String",
  "variables" : {
    "String" : "String"
  },
  "documentationVersion" : "String"
}

9. POST deployment:create http://192.168.3.44:4567/restapis/A-Z63A-Z075688/deployments

{
  "stageName" : "PatentApi",
  "stageDescription" : "PatentApi",
  "description" : "PatentApi"
}

10. GET Try the api gateway url http://192.168.3.44:4567/restapis/A-Z63A-Z075688/PatentApi/user_request/title But get

{
    "message": "API Gateway endpoint \"/restapis/A-Z63A-Z075688/PatentApi/_user_request_/title\" for method \"GET\" not found"
}

[Waldemar Hummer]

Thanks @yezhongkai , the last code snippet ("API Gateway endpoint ... not found") tells me that you might not be using the latest version. Can you please pull the latest version of the Docker image (tagged with 0.5.3 and latest) and give it another try?

Also, thanks for the detailed steps and code snippets, however, for full reproducibility it would be even better to get the full list of API commands (aws --endpoint-url=... ...) or a complete script that can be simply executed to reproduce the issue. Thanks!

[Waldemar Hummer]

Closing this issue for now, assuming that it works with the latest version. If there are still any issues, please raise a new issue with detailed instructions in this repo: https://github.com/localstack/localstack Thanks.