Building a microservice for image super-scaling

Super-scaling, neural enhancement, super resolution, AI upscaling — various terms that represent the same idea. You can use an artificial neural network to upscale images, instead of regular bilinear/bicubic algorithms. Buuuuut, that’s not really what this article is about…

Left — Original, Right — Image Super-Resolution

This tutorial intends to present an example of how you can wrap any AI related solutions (that require you to stick with Python) in a convenient stateless service, that can be easily deployed, scaled, and used anywhere in your architecture.

What we will build today:

Base microservice in Python, with Flask framework
Endpoint for image upscaling via ISR package (ISR GitHub)
Docker image for our microservice
REST API client in Java to communicate with our microservice

As always — final code and sample images for this tutorial can be found in my GitHub repo by the link at the end of this article.

Part 1 —Basic Flask app

Let’s start by making a simple Flask application, a basic microservice in Python, with one ping endpoint in it.

First we’re gonna need a JSON object message_protocol/ping_output.py to return data from this endpoint

1import json
2
3class PingOutput:
4    def __init__(self, success: bool):
5        self.success = success
6
7    def toJSON(self):
8        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=True, indent=4)

Then let’s make the endpoint logic have its own class, and put it in resource_ping.py, I know, looks excessive, but trust me it pays off when you have some real workload in these endpoints.

1from message_protocol.ping_output import PingOutput
2
3class ResourcePing:
4    def main(self) -> PingOutput:
5        return PingOutput(True)

Make sure that the resource is instantiated only once (on app startup), by adding a separate script resources.py

1# Make sure all resources are instantiated here
2from resource_ping import ResourcePing
3resourcePing = ResourcePing()

Now the main piece — a Flask app that maps /ping URL into our resource. Let’s name it flask_app.py

1from flask import Flask
2from flask import request
3from flask import Response
4from resources import resourcePing
5
6app = Flask(__name__)
7
8@app.route('/ping', methods=['GET'])
9def ping():
10    output = resourcePing.main()
11    json = output.toJSON()
12    return Response(json, mimetype='application/json')

With that in place you can run the service in terminal via

1export FLASK_APP=flask_app.py
2flask run --host=0.0.0.0 --port=7777

And test the response by doing GET http://localhost:7777/ping

The backbone of our Python microservice is now completed, and we can add new resources in a similar manner.

Part 2 — Integrate ISR

With the basic Flask app up and running we can add an endpoint for image upscaling, but first let’s take a look at how Image Super-Resolution works by doing a quick test outside the Flask application. You can find more detailed ISR usage examples on their GitHub, I’ll just provide a quick sample for our case. We will use the Artefact Cancelling GANS model in this tutorial.

1import numpy as np
2from PIL import Image
3from ISR.models import RDN
4
5inputPath = '../../sample-images/t4.jpg'
6outputPath = '../../sample-images/t4_up.jpg'
7imagePil = Image.open(inputPath)
8imageNumpy = np.array(imagePil)
9model = RDN(weights='noise-cancel')
10scaledNumpy = model.predict(imageNumpy)
11scaledPil = Image.fromarray(scaledNumpy)
12scaledPil.save(outputPath)

Left — Original, Right — Image Super-Resolution

Now let’s put this scaling into a new resource. To pass images in JSON objects I will use base64 string image encoding, which simplifies this tutorial, but a more reasonable approach for real world applications would be passing only URLs to images.

We’ll define input/output JSON objects (although their contents are the same, I will use 2 different objects here, as in real applications we will probably want to pass additional data in them)

1import json
2
3class UpscaleInput:
4    def __init__(self, imageBase64: str):
5        self.imageBase64 = imageBase64
6
7    def toJSON(self):
8        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=False, indent=4)
9
10def parseUpscaleInput(dictionary) -> UpscaleInput:
11    imageBase64 = dictionary['imageBase64']
12    return UpscaleInput(imageBase64)
13
14class UpscaleOutput:
15    def __init__(self, imageBase64: str):
16        self.imageBase64 = imageBase64
17
18    def toJSON(self):
19        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=False, indent=4)
20
21def parseUpscaleOutput(dictionary) -> UpscaleOutput:
22    imageBase64 = dictionary['imageBase64']
23    return UpscaleOutput(imageBase64)

And now the resource itself, resource_upscale.py

1import io
2import base64
3import numpy as np
4from PIL import Image
5from ISR.models import RDN
6from message_protocol.upscale_input import UpscaleInput
7from message_protocol.upscale_output import UpscaleOutput
8
9MODEL = RDN(weights='noise-cancel')
10
11class ResourceUpscale:
12    def main(self, upscaleInput: UpscaleInput) -> UpscaleOutput:
13        # Parse base64 string into bytes array
14        inputImageBytesArray = base64.b64decode(upscaleInput.imageBase64)
15
16        # Open the image
17        imagePil = Image.open(io.BytesIO(inputImageBytesArray))
18        imageNumpy = np.array(imagePil)
19
20        # Scale the image
21        scaledNumpy = MODEL.predict(imageNumpy)
22        scaledPil = Image.fromarray(scaledNumpy)
23
24        # Write scaled image as bytes array
25        outputImageBytesArrayIO = io.BytesIO()
26        scaledPil.save(outputImageBytesArrayIO, format=imagePil.format, quality=100)
27        outputImageBytesArray = outputImageBytesArrayIO.getvalue()
28
29        # Convert back to base64 string
30        outputImageBase64 = base64.b64encode(outputImageBytesArray).decode('utf-8')
31        return UpscaleOutput(outputImageBase64)

Don’t forget to instantiate it in resources.py

1# Make sure all resources are instantiated here
2from resource_ping import ResourcePing
3from resource_upscale import ResourceUpscale
4resourcePing = ResourcePing()
5resourceUpscale = ResourceUpscale()

And map the /upscale URL in flask_app.py

1from flask import Flask
2from flask import request
3from flask import Response
4from resources import resourcePing, resourceUpscale
5from message_protocol.upscale_input import parseUpscaleInput
6
7app = Flask(__name__)
8
9@app.route('/ping', methods=['GET'])
10def ping():
11    output = resourcePing.main()
12    json = output.toJSON()
13    return Response(json, mimetype='application/json')
14
15@app.route('/upscale', methods=['POST'])
16def upscale():
17    input = parseUpscaleInput(request.json)
18    output = resourceUpscale.main(input)
19    json = output.toJSON()
20    return Response(json, mimetype='application/json')

At this stage let’s leave the Python side of things, and switch to building our environment and client to properly run and communicate with this service.

Part 3 — Docker image

Our Docker image will be pretty straight-forward, we will start with a base Ubuntu image, install Python, PIP and all the dependencies needed to run this service in a container, copy the source code and run the service on container launch. So the Dockerfile ends up looking like this:

1FROM ubuntu:20.04
2
3# Initial setup & install system utils
4RUN apt-get update && apt-get upgrade -y
5RUN apt-get install -y apt-utils software-properties-common
6RUN apt-get install -y vim wget git
7
8# Install Python 3.8 & Pip 3
9WORKDIR /
10RUN add-apt-repository -y ppa:deadsnakes/ppa
11RUN apt-get install -y python3.8
12RUN echo "alias python='python3.8'" >> ~/.bashrc
13RUN echo "alias python3='python3.8'" >> ~/.bashrc
14RUN apt-get install -y python3-pip
15RUN echo "alias pip='pip3'" >> ~/.bashrc
16
17# Install Python dependencies
18RUN pip3 install flask
19RUN pip3 install numpy
20RUN pip3 install tensorflow
21RUN pip3 install ISR --ignore-installed tensorflow
22
23# Copy source code
24RUN mkdir /python-service
25COPY . /python-service
26WORKDIR /python-service
27RUN chmod a+x run.sh
28
29# Run
30CMD './run.sh'

As for the run.sh script

1#!/bin/sh
2export FLASK_APP=flask_app.py
3flask run --host=0.0.0.0 --port=7777

Now you can build and run the Docker container from terminal

1docker build -t service-ai-image . && docker run --rm -ti -p 7777:7777 --name service-ai-container service-ai-image

And test that it’s pinging by doing GET http://localhost:7777/ping

Part 4 — Java client

Probably the easiest part of this tutorial — we now need to build a REST API client that communicates with our service. I’m coding this in Java, but this part should be doable in any other language you prefer. I’m gonna use Lombok annotations to speed things up and reduce boilerplate code, Gson for easier JSON mappings, and Apache HTTP client to call the service. Then, of course, JUnit for testing.

JSON objects would look like this

1public interface JsonSerializable extends Serializable {
2    default String toJson() {
3        return new GsonBuilder().create().toJson(this);
4    }
5}
6
7@Getter @AllArgsConstructor @NoArgsConstructor @ToString @EqualsAndHashCode
8public class PingOutput implements JsonSerializable {
9    private boolean success;
10}
11
12@Getter @AllArgsConstructor @NoArgsConstructor @ToString @EqualsAndHashCode
13public class UpscaleInput implements JsonSerializable {
14    private String imageBase64;
15
16    public UpscaleInput(byte[] imageBytesArray) {
17        this(Base64.getEncoder().encodeToString(imageBytesArray));
18    }
19
20    public UpscaleInput(File imageFile) throws IOException {
21        this(Files.readAllBytes(imageFile.toPath()));
22    }
23}
24
25@Getter @AllArgsConstructor @NoArgsConstructor @ToString @EqualsAndHashCode
26public class UpscaleOutput implements JsonSerializable {
27    private String imageBase64;
28
29    public byte[] toBytesArray() {
30        return Base64.getDecoder().decode(imageBase64);
31    }
32}

We also need a helper class with some logic about handling HTTP requests (in real world apps you’d want some more try-catch blocks here, some decent error handling logic, but this works fine for our case)

1public class ClientUtils {
2    private static String getJsonResponse(HttpResponse httpResponse) throws Exception {
3        if (httpResponse.getStatusLine().getStatusCode() == 200) {
4            HttpEntity httpEntity = httpResponse.getEntity();
5            return EntityUtils.toString(httpEntity);
6        } else {
7            throw new IllegalStateException("Http response status code is invalid: " + httpResponse.getStatusLine().getStatusCode() + ", " + EntityUtils.toString(httpResponse.getEntity()));
8        }
9    }
10
11    public static <M> M parseJsonResponse(String jsonResponse, Type responseResultType) throws Exception {
12        return new GsonBuilder().create().fromJson(jsonResponse, responseResultType);
13    }
14
15    public static String doGet(String url) throws Exception {
16        HttpClient httpClient = HttpClientBuilder.create().build();
17        HttpGet httpGet = new HttpGet(url);
18        httpGet.setHeader("Accept", "application/json");
19        return getJsonResponse(httpClient.execute(httpGet));
20    }
21
22    public static String doPost(String url, String jsonPayload) throws Exception {
23        HttpClient httpClient = HttpClientBuilder.create().build();
24        HttpPost httpPost = new HttpPost(url);
25        httpPost.setEntity(new StringEntity(jsonPayload, ContentType.APPLICATION_JSON));
26        httpPost.setHeader("Accept", "application/json");
27        httpPost.setHeader("Content-type", "application/json");
28        return getJsonResponse(httpClient.execute(httpPost));
29    }
30}

Now we can finally write the main client itself, with 2 methods — ping and upscale.

1public class Client {
2    private static String BASE_URL = "http://localhost:7777";
3
4    public static PingOutput ping() throws Exception {
5        String url = BASE_URL + "/ping";
6        String response = ClientUtils.doGet(url);
7        return ClientUtils.parseJsonResponse(response, PingOutput.class);
8    }
9
10    public static UpscaleOutput upscale(UpscaleInput input) throws Exception {
11        String url = BASE_URL + "/upscale";
12        String response = ClientUtils.doPost(url, input.toJson());
13        return ClientUtils.parseJsonResponse(response, UpscaleOutput.class);
14    }
15}

It’s time to test the whole puzzle together! Make sure the service is running in Docker, and launch these tests

1@Ignore
2public class ClientTest {
3    @Test
4    public void ping() throws Exception {
5        PingOutput output = Client.ping();
6        assertNotNull(output);
7        assertTrue(output.isSuccess());
8    }
9
10    @Test
11    public void upscale() throws Exception {
12        String inputFilePath = "../sample-images/t7.jpg";
13        String outputFilePath = "./sample-images/t7_up.jpg";
14        File inputFile = new File(inputFilePath);
15        UpscaleInput input = new UpscaleInput(inputFile);
16        UpscaleOutput output = Client.upscale(input);
17        assertNotNull(output);
18        assertFalse(output.getImageBase64().isEmpty());
19        File outputFile = new File(outputFilePath);
20        FileUtils.writeByteArrayToFile(outputFile, output.toBytesArray());
21    }
22}

If everything went well you should have a scaled image saved on your drive

Left — Original, Right — Image Super-Resolution

That’s it! Thank you for following through with this tutorial, and a quick recap before we go — we’ve briefly examined image super-scaling, built an AI microservice as a Flask app in Python, wrapped it in a Docker container, and tested it though REST API client in Java. I hope this article was helpful and I managed to show you how to integrate Python AI services in your project.

GitHub repo with source code for this tutorial

In case you’d like to check my other work or contact me:

Building a microservice for image super-scaling

Part 1 —Basic Flask app

Part 2 — Integrate ISR

Part 3 — Docker image

Part 4 — Java client

More articles from TekLeo

How to automatically deskew (straighten) a text image using OpenCV

License plate removal with OpenCV