Research Notes2: Model Persistence Issue in Machine Learning
Today I met this problem and had to slow down my work to rerun some of my models to make sure everything works as expected when I tried to train the machine learning model. The problem is that when I trained the neural network, I forgot to save the preprocessing results like the
MinMaxScaler() I used, so this caused a huge problem that I had to retrain everything to make sure the model works correctly.
Deploy Machine Learning Models
Although this is not an error, it is a big pitfall in your typical training of machine learning. The typical machine learning course tends to focus on the algorithm and mathematical reasoning behind the model, it does not tell you how to deploy the model. This is reasonable because actually nowadays, there are no golden rules about how to deploy machine learning models.
Deploying a machine learning model is still at its stone ages. The problem is even harder to solve if you take the software dependency issues into consideration. For example, after you trained a good machine learning model, how could you provide it to the public? Assuming there are \(10^6\) users who want to use your service, how could you optimize the codes to reduce the latency and computing workload? How to help the user input the correct format of their own data? Is there any method to deal with vicious intention users?
Actually, if we want to go deeper into the ideas of deploying your machine learning model, it will go to the area of computer systems which is rapidly changing and actively researched areas of computer science. Taking one example, I learned one course in UW-Madison that is taught by Prof. Shivaram Venkataraman in Fall 2018, CS 744 Big Data Systems - UW Madison, Fall 2018, you can find an entire section goes to how to deploy machine learning models online, e.g. Ray: A Distributed Framework for Emerging AI Applications.
Check out the articles interested to you.
So go back to this article, I want to talk about model persistence and how to make sure you have gotten everything you needed. Generally, it is about tips to help you save the training results successfully.
So three things I think we need to store are:
Neural Network Results
For preprocessing results, I think we can use
joblib to save models.
Below is a code demos from 3.4. Model persistence
from sklearn import svm from sklearn import datasets clf = svm.SVC() X, y= datasets.load_iris(return_X_y=True) clf.fit(X, y) # Using Pickle import pickle s = pickle.dumps(clf) clf2 = pickle.loads(s) clf2.predict(X[0:1])
Another example about using
from joblib import dump, load # Save Model dump(clf, 'filename.joblib') # Load Saved Model clf = load('filename.joblib')
Neural Network Results
This part is typical, there are two key things you need to keep in mind, one is the structure of your neural network, another one is the weights of your neural network. Structure is usually stored in
json format since it is a layered data structure and weight is usually stored in
h5 format since it is pure number and typically is large.
Code demos of saving models and commented last section is about how to load saved models
# construct the argument parser and parse the arguments this will be needed for saving results in a different folder ap = argparse.ArgumentParser() ap.add_argument("-s", "--savedResults", type=str, required=True, help="path to save results of both model and weights") # create results file when needed if not os.path.exists(args["savedResults"]): os.makedirs(args["savedResults"]) # serialize model to JSON model_json = model.to_json() with open(args["savedResults"] + "/model_structure.json", "w") as json_file: json_file.write(model_json) # serialize weights to HDF5 model.save_weights(args["savedResults"]+"/model_weights.h5") print("Saved model to disk") # In case needs to reload the model """ # load json and create model json_file = open('model.json', 'r') loaded_model_json = json_file.read() json_file.close() loaded_model = model_from_json(loaded_model_json) # load weights into new model loaded_model.load_weights("model.h5") print("Loaded model from disk") """
Following the same logic in saving neural network training results, we will use the following code to save training logs.
# construct the argument parser and parse the arguments this will be needed for saving results in a different folder ap = argparse.ArgumentParser() ap.add_argument("-l", "--trainingLog", type=str, required=True,help="path to save training logs and metrics curves") if not os.path.exists(args["trainingLog"]): os.makedirs(args["trainingLog"]) # train the model print("[INFO] training model...") history = model.fit(X_train, y_train_scaled, validation_data=(X_test, y_test_scaled), epochs=300, batch_size=64) # Plot the training Processing # https://keras.io/visualization/ # list all data in history print(history.history.keys()) # Plot training & validation loss values plt.plot(history.history['loss']) plt.plot(history.history['val_loss']) plt.title('Model loss') plt.ylabel('Loss') plt.xlabel('Epoch') plt.legend(['Train', 'Validation'], loc='upper right') plt.savefig(args["trainingLog"]+"/loss.png", dpi=150) plt.close()