Skip to content

image-processing-pipeline.md

Latest commit

 

History

History
264 lines (186 loc) · 8.91 KB

image-processing-pipeline.md

File metadata and controls

264 lines (186 loc) · 8.91 KB

Image Processing Pipeline

In this sample, we'll build an image processing pipeline to connect Google Cloud Storage events to various services with Knative Eventing on GKE.

Image Processing Pipeline

  1. An image is saved to an input Cloud Storage bucket.
  2. Cloud Storage update event is read into Knative by CloudStorageSource.
  3. Filter service receives the Cloud Storage event. It uses Vision API to determine if the image is safe. If so, it creates a custom CloudEvent of type dev.knative.samples.fileuploaded and passes it onwards.
  4. Resizer service receives the fileuploaded event, resizes the image using ImageSharp library, saves to the resized image to the output bucket, creates a custom CloudEvent of type dev.knative.samples.fileresized and passes the event onwards.
  5. Watermark service receives the fileresized event, adds a watermark to the image using ImageSharp library and saves the image to the output bucket.
  6. Labeler receives the fileuploaded event, extracts labels of the image with Vision API and saves the labels to the output bucket.

Prerequisites

We're assuming that you already went through Cloud Storage triggered service tutorial where you setup Knative with GCP & PubSub Topic and also initialized Cloud Storage with Pub/Sub events. Here we will start with creating buckets for the pipeline.

Create storage buckets

Create 2 unique storage buckets to save pre and post processed images:

export BUCKET1="$(gcloud config get-value core/project)-images-input"
export BUCKET2="$(gcloud config get-value core/project)-images-output"
gsutil mb gs://${BUCKET1}
gsutil mb gs://${BUCKET2}

CloudStorageSource

Create a CloudStorageSource to connect storage events from the first bucket to the Broker in Knative Eventing.

cloudstoragesource.yaml defines the CloudStorageSource. Make sure you update the bucket name to the actual bucket name in your project.

Create the CloudStorageSource:

kubectl apply -f cloudstoragesource.yaml

You should see the CloudStorageSource running:

kubectl get cloudstoragesource

NAME                         READY   REASON   AGE
storagesource-images-input   True             23s

Broker

Make sure there's a Broker in the default namespace by following instructions in Broker Creation page.

Enable Vision API

Some services use Vision API. Make sure the Vision API is enabled:

gcloud services enable vision.googleapis.com

Filter

This service receives Cloud Storage events for saved images. It uses Vision API to determine if the image is safe. If so, it passes a custom event onwards.

Service

The code of the service is in filter folder.

Inside the top level processing-pipelines folder, build and push the container image:

export SERVICE_NAME=filter
docker build -t meteatamel/${SERVICE_NAME}:v1 -f image/${SERVICE_NAME}/csharp/Dockerfile .
docker push meteatamel/${SERVICE_NAME}:v1

Create the service defined in kservice.yaml:

kubectl apply -f kservice.yaml

Trigger

The trigger of the service filters on Cloud Storage finalize events: com.google.cloud.storage.object.finalize.

Create the trigger for the service defined in trigger.yaml:

kubectl apply -f trigger.yaml

Resizer

This service receives the custom event, resizes the image using ImageSharp library and passes the event onwards.

Service

The code of the service is in resizer folder.

Inside the top level processing-pipelines folder, build and push the container image:

export SERVICE_NAME=resizer
docker build -t meteatamel/${SERVICE_NAME}:v1 -f image/${SERVICE_NAME}/csharp/Dockerfile .
docker push meteatamel/${SERVICE_NAME}:v1

Create the service defined in kservice.yaml. Make sure you update the BUCKET env variable to the value of $BUCKET2:

kubectl apply -f kservice.yaml

Trigger

The trigger of the service filters on dev.knative.samples.fileuploaded event types which is the custom event type emitted by the filter service.

Create the trigger for the service defined in trigger.yaml:

kubectl apply -f trigger.yaml

Watermark

This service receives the event, adds the watermark to the image using ImageSharp library and saves the image to the output bucket.

Service

The code of the service is in watermarker folder.

Inside the top level processing-pipelines folder, build and push the container image:

export SERVICE_NAME=watermarker
docker build -t meteatamel/${SERVICE_NAME}:v1 -f image/${SERVICE_NAME}/csharp/Dockerfile .
docker push meteatamel/${SERVICE_NAME}:v1

Create the service defined in kservice.yaml. Make sure you update the BUCKET env variable to the value of $BUCKET2:

kubectl apply -f kservice.yaml

Trigger

The trigger of the service filters on dev.knative.samples.fileresized event types which is the custom event type emitted by the resizer service.

Create the trigger for the service defined in trigger.yaml:

kubectl apply -f trigger.yaml

Labeler

Labeler receives the event, extracts labels of the image with Vision API and saves the labels to the output bucket.

Service

The code of the service is in labeler folder.

Inside the top level processing-pipelines folder, build and push the container image:

export SERVICE_NAME=labeler
docker build -t meteatamel/${SERVICE_NAME}:v1 -f image/${SERVICE_NAME}/csharp/Dockerfile .
docker push meteatamel/${SERVICE_NAME}:v1

Create the service defined in kservice.yaml. Make sure you update the BUCKET env variable to the value of $BUCKET2:

kubectl apply -f kservice.yaml

Trigger

The trigger of the service filters on dev.knative.samples.fileuploaded event types which is the custom event type emitted by the filter service.

Create the trigger for the service defined in trigger.yaml:

kubectl apply -f trigger.yaml

Test the pipeline

Before testing the pipeline, make sure all the triggers are ready:

kubectl get trigger

NAME                  READY   REASON   BROKER    SUBSCRIBER_URI                                 AGE
trigger-filter        True             default   http://filter.default.svc.cluster.local        3d19h
trigger-labeler       True             default   http://labeler.default.svc.cluster.local       3d
trigger-resizer       True             default   http://resizer.default.svc.cluster.local       3d19h
trigger-watermarker   True             default   http://watermarker.default.svc.cluster.local   3d

You can upload an image to the input storage bucket:

gsutil cp  ../../pictures/beach.jpg gs://${BUCKET1}

After a minute or so, you should see resized, watermarked and labelled image in the output bucket:

gsutil ls gs://${BUCKET2}

gs://knative-atamel-images-output/beach-400x400-watermark.jpeg
gs://knative-atamel-images-output/beach-400x400.png
gs://knative-atamel-images-output/beach-labels.txt