Machine Learning integration with DevOps

This project aims to automate the ML process by tweaking the hyper-parameters to increase the accuracy of the model by integrating Machine Learning with DevOps tools.

A small project to automate hyper-parameters of CNN model according to model accuracy can be as follows-

1. Create container image that has Python3 and all the necessary libraries installed for development of a CNN model using Dockerfile

Dockerfile

here, I have used pre created “tensorflow/tensorflow” image from dockerhub for tensorflow environment and have created a file- requirement.txt which has all the required libraries

When we launch this image, it automatically starts to train the model in the container.

2. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins

• Job1 : Pull the Github repo automatically when some developers push the repository to Github.

Job 1 configuration in Jenkins is as follows:

This job pulls code of our CNN model from github repository provided by the user

• Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the software required for the cnn processing).

job2 triggered by job1

This job launches docker container by detecting the type of model that the developer has created

This job is only created when job1 is successfully built

• Job3 : Train your model and predict accuracy or metrics, if its accuracy is less than 80% , then tweak the machine learning model architecture.

job3 gets built only after job2

Checking accuracy

launching job4 if job3 fails, job5 if jo3 is success

Job3 checks the accuracy of the model — if the accuracy is above 80% it is considered as successful build else, the job is failed

On successful build it triggers Job5

On failed build it triggers Job4

• Job4: Retrain the model or notify that the best model is being created

job2 is triggered by job4

This job tweaks the hyper-parameters of the CNN model to increase the accuracy then job2 trains the new model again to check the accuracy

This pipeline is repeated till the accuracy crosses 80%

Build Pipeline of the project