A Blog About D4T4 & M47H

Building and Deploying a Deep Learning Model Part 3: Deploying a Serverless Microservice

16 September ’18

This is part 3 in a 3-part series (part 1, part 2) on building and deploying a deep learning model for the popular ACL 2011 IMDB dataset. In this part, I host the model on Cloud ML Engine and make it accessible via a simple HTTP Cloud Function. Give it a try!


Upload Model to Cloud ML Engine


MODEL_TIMESTAMP=$(ls -t exports/ | head -1)


gsutil rsync -c -d -r exports/$MODEL_TIMESTAMP $DEPLOYMENT_SOURCE

gcloud ml-engine models create $MODEL_NAME

gcloud ml-engine versions create $MODEL_VERSION --model $MODEL_NAME --origin $DEPLOYMENT_SOURCE \
    --python-version 2.7 --runtime-version 1.9

NOTE: Make sure the Python environment in which you build your model matches the serving environment in Cloud ML!

Expose Model with a Cloud Function

# gets predictions from cloud ml engine
def classify_movie_reviews(request):
    import flask
    import json
    import re
    import math
    import googleapiclient.discovery
    import google.auth

    headers = {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': 'POST',
        'Access-Control-Allow-Headers': 'Content-Type'

    # handle pre-flight options request
    if request.method == 'OPTIONS':
        return flask.make_response(('', 204, headers))

    _, project = google.auth.default()

    request_json = request.get_json()

    # this pulls out our proper nouns and treats them as single words
    def preprocessing(review):
        proper = r"([A-Z]([a-z]+|\.)(?:\s+[A-Z]([a-z]+|\.))*(?:\s+[a-z][a-z\-]+){0,2}\s+[A-Z]([a-z]+|\.)(?:\s+[0-9]+)?)"
        space_between_brackets = r"[\.\s]+(?=[^\[\]]*]])"
        brackets = r"(?:[\[]{2})(.*?)(?:[\]]{2})"

        review = re.sub(proper, '[[\\1]]', review)
        review = re.sub(space_between_brackets, '~', review)
        review = re.sub(brackets, '\\1', review)
        return review

    model = 'movie_reviews'
    version = request_json['version']
    instances = [preprocessing(i) for i in request_json['instances']]

    service ='ml', 'v1')
    name = 'projects/{}/models/{}/versions/{}'.format(project, model, version)

    response = service.projects().predict(
        body={'instances': instances}

    if 'error' in response:
        raise RuntimeError(response['error'])

    # clear out nan if they exist
    for r in response['predictions']:
        if all([math.isnan(i) for i in r['prob']]):
            r['prob'] = []
            r['class'] = -1

    return flask.make_response((

NOTE: Additional preprocessing for grouping movie names and proper nouns is replicated here since it could not be embedded in the TF input serving function.

Link to all code: