Convolutional Neural Network (CNN) ¶
Project: Implementing a Dog Breed Identification Algorithm App ¶
In this notebook file, some template code is already available to you, but you also need to implement more features to complete the project. You do not need to modify any of the given code unless explicitly requested. The title starting with '(practice)' indicates that you have the functionality you need to implement in the next section of the code. These sections are accompanied by detailed instructions, and the parts that need to be implemented are also marked with 'TODO' in the comments. Please read all the tips carefully.
In addition to implementing the code, you also need to answer some questions related to the project and code. Every question that needs to be answered will be marked with 'Question X' . Please read each question carefully and write the full answer in the 'Answer' section of the question. We will rate the items you submit based on your answers to the questions and the functionality of the code implementation.
Tip: The Code and Markdown areas can be run with the Shift + Enter shortcut. In addition, Markdown can enter edit mode by double-clicking.
The part of the project that shows _selected _ can help your project stand out, rather than just meeting the minimum requirements. If you decide to pursue a higher challenge, please complete the _select_part of the code in this notebook.
Let's get started ¶
In this notebook, you will take the first step to developing algorithms that can be used as part of a mobile or web application. At the end of this project, your program will be able to take any image provided by the user as input. If a dog can be detected from the image, it will output a prediction for the dog breed. If the image is a human face, it predicts the kind of dog that is most similar to it. The image below shows the possible output after completing the project. (...actually we want each student's output to be different!)
In the real world, you need to piece together a series of models to accomplish different tasks; for example, the algorithm used to predict dog species will be different from the algorithm that predicts humans. In the process of doing the project, you may encounter many failed predictions because there are no perfect algorithms and models. The imperfect solution you finally submit will definitely bring you an interesting learning experience!
Project content¶
We have divided this notebook into different steps, you can use the link below to browse this notebook.
- Step 0 : Import the data set
- Step 1 : Detecting faces
- Step 2 : Detecting dogs
- Step 3 : Create a CNN from scratch to classify dog breeds
- Step 4 : Use a CNN to distinguish the breed of the dog (using migration learning)
- Step 5 : Create a CNN to classify the dog breed (using migration learning)
- Step 6 : Complete your algorithm
- Step 7 : Test your algorithm
The following questions are included in the project:
- Question 1
- Question 2
- Question 3
- Question 4
- Question 5
- Question 6
- Question 7
- Question 8
- Question 9
- Question 10
- Question 11
Step 0: Import the data set¶
Import dog data set¶
In the code cell below, we imported a data set of dog images. We use scikit-learn library load_filesfunction to get the number of variables:
train_files,valid_files,test_files- a numpy array containing the file path of the imagetrain_targets,valid_targets,test_targets- a numpy array containing the uniquely encoded classification labeldog_names- the type of dog corresponding to the label consisting of a string
From sklearn.datasets import load_files
from keras.utils import np_utils
import numpy as np
from glob import glob
# define function to load train, test, and validation datasets
def load_dataset ( path ):
data = load_files ( path )
dog_files = np . array ( data [ 'filenames' ])
dog_targets = np_utils . to_categorical ( np . array ( data [ ' Target' ]), 133 )
return dog_files , dog_targets
# load train, test, and validation datasets
train_files , train_targets = load_dataset ( '/data/dog_images/train' )
valid_files , valid_targets = load_dataset ( '/data/dog_images/valid' )
test_files , test_targets = load_dataset ( '/data/dog_images /test' )
# load list of dog names
dog_names = [ item [ 20 : - 1 ] for item in sorted ( glob ( "/data/dog_images/train/*/" ))]
# print statistics about the dataset
print ( 'There are %d total dog categories.' % len ( dog_names ))
print ( 'There are %s total dog images. \n ' % len ( np . hstack ([ train_files , valid_files , Test_files ])))
print ( 'There are %d training dog images.' % len ( train_files ))
print ( 'There are %d Validation dog images.' % len ( valid_files ))
print ( 'There are %d test dog images.' % len ( test_files ))
Import face data set¶
In the code unit below, we introduced into the facial image data set, the path is stored in a file named human_filesarray of numpy.
Import random
random . seed ( 8675309 )
#Load the file name of the corrupted face dataset human_files = np . array ( glob ( "/data/lfw/*/*" ))
random . shuffle ( human_files )
# Print dataset data
print ( 'There are %d total human images.' % len ( human_files ))
Step 1: Detecting face ¶
We will use Haar feature-based cascade classifiers in OpenCV to detect faces in images. OpenCV provides a number of pre-trained face detection models that are stored in github as XML files . We have downloaded one detection model, and stores it in the haarcascadesdirectory.
In the following code unit, we will demonstrate how to use this detection model to find faces in the sample image.
Import cv2
import matplotlib.pyplot as plt
% matplotlib inline
# Extract the pre-trained face detection model
face_cascade = cv2 . CascadeClassifier ( ' haarcascades /haarcascade_frontalface_alt.xml' )
# Load color (channel order is BGR) image
img = cv2 . imread ( human_files [ 3 ])
# Grayscale the BGR image
gray = cv2 . cvtColor ( img , cv2 . COLOR_BGR2GRAY )
# Find the
faces in the image faces = face_cascade . detectMultiScale ( gray )
# Print the number
of faces detected in the image print ( 'Number of faces detected:' , len ( faces ))
# Get the recognition box
for each detected face for ( x , y , w , h ) in faces :
# Draw the recognition frame
cv2 in the face image . rectangle ( img ,( x , y ),( x + w , y + h ), ( 255 , 0 , 0 ), 2 )
# Convert BGR image to RGB image to print
cv_rgb = cv2 . cvtColor ( img , cv2 . COLOR_BGR2RGB )
# Show image with the identification box
plt . imshow ( cv_rgb )
plt . show ()
Converting an image to a grayscale image is a common process before using any of the inspection models. detectMultiScaleFunction is used is stored in face_cascadethe data input gray image are classified.
In the code above faces, the recognized face information is saved as a numpy array. Each of the lines represents a detected face. The data includes the following four information: the first two elements x, ythe x and y coordinates representing the upper left corner of the recognition frame (refer to the figure above, pay attention to the direction of the y coordinate and our default direction). different); the element represents two identification frame extending in both directions x and y axis length w, and d.
Write a face recognizer¶
We can wrap this program as a function. The input of this function is the path of the face image . When the image contains a face, the function returns True, and vice versa False. The function definition is as follows.
# If the image represented by the img_path path detects a face, return "True"
def face_detector ( img_path ):
img = cv2 . imread ( img_path )
gray = cv2 . cvtColor ( img , cv2 . COLOR_BGR2GRAY )
faces = face_cascade . detectMultiScale ( gray )
return Len ( faces ) > 0
[Exercise] Evaluate the face detection model¶
Question 1: ¶
In the bottom of the block, a face_detectorfunction is calculated:
human_filesOf the top 100 images, what is the ratio of images that can detect faces ?dog_filesOf the top 100 images, what is the ratio of images that can detect faces ?
Ideally, the probability of detecting a face in a person's image should be 100%, and the probability of detecting a face in a dog image should be 0%. You will find that our algorithm is not perfect, but the results are still acceptable. We extract the file paths of the first 100 images from each data set and store them in the human_files_shortsum dog_files_short.
NeuralNetClassifierhuman_files_short = human_files [: 100 ]
dog_files_short = train_files [: 100 ]
## Do not modify the code above
## TODO: dog_files_short based human_files_short and
test image ## in the performance of face_detector
Human = 0
Dog = 0
for IMG in NeuralNetClassifierhuman_files_short :
IF face_detector ( IMG ): Human = Human + . 1
Print ( "-> Detected {} Images of Human " . format ( human ))
for img in dog_files_short :
if face_detector ( img ): dog = Dog + 1
print ( "-> Detected {} images of dog" . format ( dog ))
Question 2: ¶
As far as the algorithm is concerned, the key to the success of the algorithm is whether the user can provide a face image with clear facial features. So, do you think that such a requirement is reasonable for the user in actual use? If you feel unreasonable, can you think of a way to detect a face even if there are no clear facial features in the image?
Answer: It does not Sence Because the make at The Machine. Absolutely CAN BE Within last of Better in Detecting Human Faces We CAN use the Add Image Augmentation to Random Noise in at The Picture, Image Making at The Vague to the Solve the this problem...
Optional: ¶
We recommend using opencv's face detection model in your algorithm to detect human images, but you are free to explore other methods, especially try to use deep learning to solve it :). Please use the code unit below to design and test your face monitoring algorithm. If you decide to complete this _select _ task, you need to report the performance of the algorithm on each data set.
## (Optional) TODO: Report the performance of another face detection algorithm on the LFW dataset
### You can use the required number of code units at will.
Step 2: Detect dog licking
In this section, we use the pre-trained ResNet-50 model to detect dogs in the image. The first line of code below is to download the network structure parameters of the ResNet-50 model and the pre-training weights based on the ImageNet data set.
ImageNet This is currently a very popular data set that is often used to test algorithms related to computer vision tasks such as image classification. It contains more than 10 million URLs, each linked to an image of an object corresponding to 1000 categories . When you enter an image, the ResNet-50 model returns a prediction of the object in the image.
From keras.applications.resnet50 import ResNet50
# Define ResNet50 model
ResNet50_model = ResNet50 ( weights = ' imagenet ' )
Data preprocessing ¶
- When using TensorFlow as the backend, in Keras, the CNN input is a 4-dimensional array (also known as a 4-dimensional tensor) with dimensions in dimensions
(nb_samples, rows, columns, channels). Whereinnb_samplesrepresents the total number of images (or samples),rows,columns, andchannelsthe number of lines, respectively, representing an image, the number of columns and number of channels.
- Below
path_to_tensorfunction realize a file path character string type color image as input and returns a four-dimensional tensor, as input Keras CNN. Because our input image is a color image, they have the three channels (channelsas3).- The function first reads an image and then scales it to a 224×224 image.
- Subsequently, the image is adjusted to have a tensor of 4 dimensions.
- For any input image, the dimensions of the last returned tensor are:
(1, 224, 224, 3).
paths_to_tensorThe function takes a numpy array of strings of image paths as input and returns a 4-dimensional tensor with dimensions of each dimension(nb_samples, 224, 224, 3). Here,nb_samplesis the number of samples or the number of images in the data of the provided image path. You can alsonb_samplesbe understood as a three-dimensional data set the number of tensors (each represented by a three-dimensional tensor different images.
From keras.preprocessing import image
from tqdm import tqdm
Def path_to_tensor ( img_path ):
# Load RGB images with PIL as PIL.Image.Image type
img = image . load_img ( img_path , target_size = ( 224 , 224 ))
# Convert PIL.Image.Image type to format (224, 224, 3) 3-dimensional tensor
x = image . img_to_array ( img )
# Convert the 3-dimensional tensor into a 4-dimensional tensor of the form (1, 224, 224, 3) and return
return np . expand_dims ( x , Axis = 0 )
Def paths_to_tensor ( img_paths ):
list_of_tensors = [ path_to_tensor ( img_path ) for img_path in tqdm ( img_paths )]
return np . vstack ( list_of_tensors )
Prediction based on ResNet-50 architecture ¶
For the four-dimensional tensors obtained through the above steps, some additional processing is required before inputting them to the ResNet-50 network, or other similar pre-training models in Keras:
- First, the channel order of these images is RGB, and we need to rearrange their channel order to BGR.
- Second, the input to the pre-training model is subjected to an additional normalization process. Therefore, we also want to normalize these tensors here, that is, subtract the pixel mean
[103.939, 116.779, 123.68](in RGB mode, calculated from all ImageNet images) for all pixels of all images .
Import preprocess_inputfunction implements these features. If you are very interested, you can here to see preprocess_inputthe code.
After implementing the image processing part, we can use the model to make predictions. By this step predictto implement a method, which returns an i-th vector element of a vector represents the probability that the i-th image belongs ImageNet categories. This is done by the ResNet50_predict_labelsimplemented function.
By taking the argmax function on the predicted vector (finding the subscript number with the largest probability value), we can get an integer, which is the class of the object predicted by the model. Further according to this list , we can know that which specific breed of dog.
From keras.applications.resnet50 import preprocess_input , decode_predictions
def ResNet50_predict_labels ( img_path ):
# return the predicted vector of the image of the img_path path
img = preprocess_input ( img_path ) # I cannot use directory because I don't know where udacity stores the files
# img = preprocess_input (path_to_tensor (img_path))
return NP . the argmax ( ResNet50_model . Predict ( IMG ))
Complete the dog detection model¶
In the study list , you will notice that the dog category corresponding number is 151-268. Therefore, when checking the model pre-trained dogs comprising determining whether the image, we only need to check as to ResNet50_predict_labelswhether a function returns a value between 151 and 268 (comprising a section endpoint).
We do this through these ideas below dog_detectorfunction, if it is detected from the image into the dog returns True, otherwise it returns False.
Def dog_detector ( img_path ):
prediction = ResNet50_predict_labels ( img_path )
return (( prediction <= 268 ) & ( prediction >= 151 ))
### TODO: Test the performance of the dog_detector function in human_files_short and dog_files_short
human_count = 0
dogs_count = 0
humans = paths_to_tensor ( NeuralNetClassifierhuman_files_short )
Dogs = paths_to_tensor ( dog_files_short )
for Human in humans : # (224, 244,. 3)
Human = NP . expand_dims ( Human , Axis = 0 ) # the I CAN Not use Directory Because the I do Not know WHERE Udacity
Stores the files if dog_detector ( human ): human_count = human_count + 1
for dog In dogs :
dog = np . expand_dims ( dog , axis = 0 ) # I cannot use directory because I don't know where udacity stores the files
if dog_detector ( dog ): dogs_count = dogs_count + 1
Print ( "Error: {} / {} " . format ( human_count + 1e-10 , dogs_count + 1e-10 ))
Step 3: Create a CNN from the beginning to classify the dog breed¶
Now we have implemented a function that recognizes humans and dogs in the image. But we need a further way to identify the dog's category. In this step, you need to implement a convolutional neural network to classify the breed of the dog. You need to implement your convolutional neural network from scratch (at this stage, you can't use migration learning), and you need to achieve more than 1% test set accuracy. In the five steps of this project, you also have the opportunity to use migration learning to achieve a model with greatly improved accuracy.
When adding a convolutional layer, be careful not to add too many (trainable) layers. More parameters mean longer training time, which means you are more likely to need a GPU to speed up the training process. Fortunately, Keras provides the functions needed to easily predict the time it takes to spend each iteration (epoch). You can infer the training time required for your algorithm.
It is worth noting that categorizing the image of a dog is a challenging task. Because even a normal person, it is difficult to distribute the Leytan and Welsh Springer Spaniel.
| Brittany | Welsh Springer Spaniel |
|---|---|
 |
 |
It is not difficult to find that other dog breeds have small inter-class differences (such as Golden Retriever and American Water Hound).
| Curly-Coated Retriever | American Water Spaniel |
|---|---|
 |
 |
Similarly, labradors are available in yellow, brown and black. Then the vision-based algorithm you design will have to overcome this higher inter-class difference to achieve the ability to classify these same-color dogs into the same breed.
| Yellow Labrador | Brown Labrador (Chocolate Labrador) | Black Labrador |
|---|---|---|
 |
 |
 |
We also mentioned that the random classification will give a very low result: regardless of the slight imbalance of the variety, the probability of randomly guessing the correct variety is 1/133, and the corresponding accuracy is less than 1%.
Remember that in the field of deep learning, practice is much higher than theory. Try a lot of different frameworks, trust your instincts! Of course, have fun!
Data preprocessing ¶
By dividing the pixel value of each image by 255, we normalized the image.
From PIL import ImageFile
ImageFile . LOAD_TRUNCATED_IMAGES = True
# Keras data preprocessing process
train_tensors = paths_to_tensor ( train_files ) . astype ( 'float32' ) / 255
valid_tensors = paths_to_tensor ( valid_files ) . astype ( 'float32' ) / 255
test_tensors = paths_to_tensor ( test_files ) . astype ( 'float32' ) / 255
[Training] model architecture ¶
Create a convolutional neural network to classify dog breeds. In the end, you execute a block of code model.summary()to output summary information about your model.
We have already imported some of the required Python libraries for you, and you can import them yourself if needed. If you are having trouble with the process, here is a little hint for you - the model can achieve more than 1% test accuracy in 5 epoch and can be trained quickly on the CPU.
Question 4: ¶
In the code block below, try to use Keras to build the architecture of the convolutional network and answer related questions.
- You can try to build a model of a convolutional network yourself, then you need to answer the specific steps you took to build the convolutional network (what layers are used) and why.
- You can also build a convolutional network according to the steps suggested in the figure above, so please explain why the above architecture can perform well on this issue.
Reply:
Functions of each layer:
- Conv2D: To detect spacial area
- MaxPooling: To reduce the number of weights needed (compress layer)
- Batch Normalization: Normalize the data so that the model can fit the data better (the intuition is that the normalization create more space between data pointss so that the classification can be better. although normalizations don't change the result the riotically, they do help in practice. )
- Activation(relu): relu guarantee that the zero-gradient problems don't exist. It is also zero centered to avoid zig-zag into global minimum.
- GlobalAveragePooling: the same function as MaxPooling (reduce weights needed)
- Dense: final calculation without spacial properties and classification into 133 categories.
From keras.layers import Conv2D , MaxPooling2D , GlobalAveragePooling2D , BatchNormalization , Activation
from keras.layers import Dropout , Flatten , Dense
from keras.models import Sequential
Print
model = Sequential ()
Model . the Add ( Conv2D ( 16 , ( . 3 , . 3 ), input_shape = ( 224 , 224 , . 3 ), Activation = None , padding = "Valid" ))
Model . the Add ( MaxPooling2D (( 2 , 2 )))
Model . Add ( BatchNormalization ())
model . add ( Activation ( 'relu'))
Model . The Add ( Conv2D ( 32 , ( . 3 , . 3 ), Activation = None , padding = "Valid" ))
Model . The Add ( MaxPooling2D (( 2 , 2 )))
Model . The Add ( BatchNormalization ())
Model . Add ( Activation ( 'relu' ))
model . add ( Conv2D ( 64, ( 3 , 3 ), activation = None , padding = "valid" ))
model . add ( MaxPooling2D (( 2 , 2 )))
model . add ( BatchNormalization ())
model . add ( Activation ( 'relu' ))
Model . add ( Conv2D ( 128 , ( 3 , 3 ), activation =None , padding = "valid" ))
model . add ( MaxPooling2D (( 2 , 2 )))
model . add ( BatchNormalization ( ) )
model . add ( Activation ( 'relu' ))
model . add ( GlobalAveragePooling2D ( data_format = None )))
model . add ( Dense ( 128 , activation ='relu' ))
model . add ( Dense ( 133 , activation = 'softmax' ))
### TODO: Define your network architecture
Model . summary ()
## Compile model
model . compile ( optimizer = 'rmsprop' , loss = 'categorical_crossentropy' , metrics = [ 'accuracy' ])
[Exercise] training model¶
Question 5: ¶
Train the model in the code unit below. Use model checkpointing to store the model with the lowest validation set loss.
Topics can be: You can also do the training set data enhancement , to optimize the performance of the model.
From keras.callbacks import ModelCheckpoint
### TODO: Set the number of epochs for the training model
Epochs = 5
### Do not modify the code below
checkpointer = ModelCheckpoint ( filepath = 'saved_models / weights.best.from_scratch.hdf5' ,
verbose = . 1 , save_best_only = True )
Model . fit ( train_tensors , train_targets ,
validation_data = ( valid_tensors , valid_targets ),
epochs = epochs , batch_size = 20 , callbacks = [ checkpointer ], verbose = 1 )
## Load the model with the best validation loss
Model . load_weights ( 'saved_models/weights.best.from_scratch.hdf5' )
Test model¶
Try your model on the test data set of the dog image. Ensure test accuracy is greater than 1%.
# Get the test data set index predicted image for each dog breeds
dog_breed_predictions = [ NP . The argmax ( Model . Predict ( NP . Expand_dims ( Tensor , Axis = 0 ))) for Tensor in test_tensors ]
# Test report accuracy
test_accuracy = 100 * NP . SUM ( NP . Array ( dog_breed_predictions ) == NP . The argmax ( test_targets , Axis = . 1 )) / len ( dog_breed_predictions )
Print ( 'the Test Accuracy: % .4f %% ' % Test_accuracy )
Step 4: Use a CNN to distinguish the breed of the dog¶
Using the Transfer Learning approach can help us significantly reduce training time without compromising accuracy. In the following steps, you can try to use training to train your own CNN.
Get the feature vector extracted from the image (Bottleneck Features) ¶
Bottleneck_features = np . load ( '/data/bottleneck_features/DogVGG16Data.npz' )
train_VGG16 = bottleneck_features [ 'train' ]
valid_VGG16 = bottleneck_features [ 'valid' ]
test_VGG16 = bottleneck_features [ 'test' ]
Model architecture ¶
The model uses the pre-trained VGG-16 model as a fixed image feature extractor, where the output of the last layer of the VGG-16 convolutional layer is directly imported into our model. We only need to add a global average pooling layer and a fully connected layer, where the fully connected layer uses the softmax activation function, which contains one node for each dog type.
VGG16_model = Sequential ()
VGG16_model . add ( GlobalAveragePooling2D ( input_shape = train_VGG16 . shape [ 1 :]))
VGG16_model . add ( Dense ( 133 , activation = 'softmax' ))
VGG16_model . summary ()
## Compiling the model
VGG16_model . compile ( loss = 'categorical_crossentropy' , optimizer = 'rmsprop' , metrics = [ 'accuracy' ])
## Training Model
checkpointer = ModelCheckpoint ( filepath = 'saved_models / weights.best.VGG16.hdf5' ,
verbose = . 1 , save_best_only = True )
VGG16_model . fit ( train_VGG16 , train_targets ,
validation_data = ( valid_VGG16 , valid_targets ),
epochs = 20 , batch_size = 20 , callbacks = [ checkpointer ], verbose = 1 )
## Load the model with the best validation loss
VGG16_model . load_weights ( 'saved_models/weights.best.VGG16.hdf5' )
Test model¶
Now, we can test how effective this CNN is in identifying the breed in the dog image test data set. We print out the test accuracy below.
# Get the test data set index predicted image for each dog breeds
VGG16_predictions = [ NP . The argmax ( VGG16_model . Predict ( NP . Expand_dims ( Feature , Axis = 0 ))) for Feature in test_VGG16 ]
# Test report accuracy
test_accuracy = 100 * NP . SUM ( NP . Array ( VGG16_predictions ) == NP . The argmax ( test_targets , Axis = . 1 )) / len ( VGG16_predictions )
Print ( 'the Test Accuracy: % .4f %% ' % Test_accuracy )
Use the model to predict dog breeds¶
From extract_bottleneck_features import *
DEF VGG16_predict_breed ( img_path ):
# extracted bottleneck characterized
bottleneck_feature = extract_VGG16 ( path_to_tensor ( img_path ))
# obtain prediction vector
predicted_vector = VGG16_model . Predict ( bottleneck_feature )
# Returns species model prediction dog
return dog_names [ NP . the argmax ( predicted_vector )]
Step 5: Create a CNN to classify the breed of the dog (using migration learning) ¶
Now you will use migration learning to build a CNN that will identify the dog breed from the image. Your CNN must have an accuracy of at least 60% on the test set.
In step 4, we used migration learning to create a CNN using a feature vector based on VGG-16 extraction. In this section, you must use another pre-training model to build a CNN. To make this task easier to implement, we have pre-trained several networks available in keras in advance:
- VGG-19 bottleneck features
- ResNet-50 bottleneck features
- Inception bottleneck features
- Xception bottleneck features
These files are named as:
Dog{network}Data.npz
Which {network}may be VGG19, Resnet50, InceptionV3or Xceptiona of. Choose a top network architecture, they have been stored in the directory /data/bottleneck_features/in.
[Exercise] Get the feature vector of the model¶
In the code block below, extract the training, test, and validation set corresponding to the bottleneck feature by running the code below.
bottleneck_features = np.load('/data/bottleneck_features/Dog{network}Data.npz')
train_{network} = bottleneck_features['train']
valid_{network} = bottleneck_features['valid']
test_{network} = bottleneck_features['test']### TODO: Get the bottleneck feature from another pre-trained CNN
bottleneck_features = np . load ( '/data/bottleneck_features/DogResnet50Data.npz' )
train_Resnet50 = bottleneck_features [ 'train' ]
valid_Resnet50 = bottleneck_features [ 'valid' ]
test_Resnet50 = Bottleneck_features [ 'test' ]
[Training] model architecture ¶
Create a CNN to classify dog breeds. At the end of your code unit block, output the structure of the network by running the following code:
<your model's name>.summary()
Question 6: ¶
Try Keras in the code block below to build the final network architecture, answer the steps and steps you take to implement the final CNN architecture, and describe why you used the network architecture during the migration learning process.
Reply:
How to transfer:
- (1): create a model
- (2): add GlobalAveragePooling to reduce weights between layers
- (3): add Dense layer to preform classification
Why trasnfer learning:
1. Transfer learning reduce the training time by freezing weight
2. Transfer learning reduce the computational power needed by freezing weight
3. Transfered models are easier to get to global minimum because Resnet50 is already trained on imagenet, which means it has a great ability to detect features. It is easier for our classifier to classify things on generated features by Resnet than to extract features and then classify by itself.### TODO: Define your framework
Resnet50 = Sequential ()
Resnet50 . add ( GlobalAveragePooling2D ( input_shape = train_Resnet50 . shape [ 1 :]))
Resnet50 . add ( Dense ( 133 , activation = 'softmax' ))
Resnet50 . summary ()
### TODO: Compile model
Resnet50 . compile ( loss = 'categorical_crossentropy' , optimizer = 'rmsprop' , metrics = [ 'accuracy' ])
[Exercise] training model¶
Question 7: ¶
Train your model in the code unit below. Use model checkpointing to store the model with the lowest validation set loss.
Of course, you can also carry out the training set data enhancement in order to optimize the performance of the model, but this step is not necessary.
### TODO: Training model
checkpointer = ModelCheckpoint ( filepath = 'saved_models/weights.best.Resnet50.hdf5' ,
verbose = 1 , save_best_only = True )
Resnet50 . fit ( train_Resnet50 , train_targets ,
validation_data = ( valid_Resnet50 , valid_targets ),
epochs = 20 , batch_size = 32 , callbacks = [ checkpointer ], verbose = 1 )
### TODO: Load the model weight with the best validation loss
Resnet50 . load_weights ( 'saved_models/weights.best.Resnet50.hdf5' )
### TODO: calculated on classification accuracy test set
Print ( test_Resnet50 . Shape )
Resnet50_predictions = [ NP . The argmax ( Resnet50 . Predict ( NP . Expand_dims ( Feature , Axis = 0 ))) for Feature in test_Resnet50 ]
test_accuracy = 100 * NP . SUM ( NP . Array ( Resnet50_predictions ) == NP . the argmax ( test_targets , Axis = . 1 )) / len ( Resnet50_predictions )
Print ( 'the Test Accuracy: % .4f %% ' % test_accuracy )
[Exercise] Use the model to test the breed of the dog¶
Implement a function whose input is the image path, the function is to predict the category of the corresponding image, and the output is the dog category ( Affenpinscher, Afghan_houndetc.) predicted by your model .
Similar to the simulation function in step 5, your function should contain the following three steps:
- Load image features based on selected models (bottleneck features)
- Enter image features into your model and return a prediction vector. Note that using the argmax function on this vector returns the sequence number of the dog type.
- Defined in step 0
dog_namesarray to return the corresponding dog category name.
Image feature extraction procedure used in the function can be extract_bottleneck_features.pyfound in. At the same time, they should have been imported in the previous code block. The CNN chosen your network, you can use the extract_{network}function to obtain corresponding image features, wherein {network}on behalf of VGG19, Resnet50, InceptionV3, or Xceptiona of.
Question 9: ¶
### TODO: Write a function that takes the path of the image as input
### and then returns the dog breed predicted by this model
from keras.applications.imagenet_utils import decode_predictions
from keras.models import Model
Net = ResNet50 ( weights = ' imagenet ' )
net . summary ()
intermediate_layer_model = Model ( inputs = net . input ,
outputs = net . get_layer ( "avg_pool" ) . output )
print ( train_tensors . shape )
# standard Resnet
# for x In train_tensors:
# x = preprocess_input(x.copy())
# preds = submit(x, net)
# print (preds)
Def extract_Resnet50 ( tensor ):
from keras.applications.resnet50 import ResNet50 , preprocess_input
return ResNet50 ( weights = ' imagenet ' , include_top = False ) . predict ( preprocess_input ( tensor ))
Def predict ( x ):
x = preprocess_input ( x . copy ())
x = np . expand_dims ( x , axis = 0 )
# intermediate_output = intermediate_layer_model.predict(x)
intermediate_output = extract_Resnet50 ( x )
preds = Resnet50 . predict ( intermediate_output )
OUT = NP . the argmax ( preds )
# print(decode_predictions(out, top=5))
return out
Step 6: Complete your algorithm ¶
Implement an algorithm whose input is the path of the image, which distinguishes whether the image contains a person, a dog, or both, and then:
- If a dog is detected from the image, the predicted variety is returned.
- If a person is detected from the image , the most similar dog breed is returned.
- If neither of them can be detected in the image, an error message is output.
We welcome you to write the function in humans and detect the image of the dog yourself, you can freely use to complete the above face_detectorand dog_detectorfunction. You need to use your CNN in step 5 to predict the dog breed.
The sample output of the algorithm is provided below, but you are free to design your own model!
Question 10: ¶
Complete your code in the code block below.
### TODO: Design your algorithm
### Freely use the required number of code units
from IPython.display import clear_output
import tqdm
Def get_name ( id ):
if id >= len ( dog_names ): raise Exception ( 'Where is my dog?' )
return dog_names [ id ] . split ( "." , 1 )[ 1 ]
i = 2 # running without GPU, super slow. You can adjust the test image as you like.
np . random . shuffle ( humans )
for i , x in enumerate ( humans [: i ]):
prediction = predict ( x )
name = get_name ( int ( prediction ))
clear_output ()
print ( "My name is {} : {} " . format( i , name ))
i = 2 # running without GPU, super slow. You can adjust the test image as you like.
np . random . shuffle ( dogs )
for i , x in enumerate ( dogs [: i ]):
prediction = predict ( x )
name = get_name ( int ( prediction ))
clear_output ()
print ( "My name is {} : {} " . format( i , name ))
Step 7: Test your algorithm ¶
In this section, you will try your new algorithm! What type of dog do you think the algorithm looks like? If you have a dog, can it accurately predict your dog's breed? If you have a cat, will it mistake your cat for a dog?
Question 11: ¶
Write code below and test your algorithm with at least 6 real-world images. You can use any photo, but please use at least two human images (with the consent of the parties) and two dogs. Please also answer the following questions:
- Is the output better than you expected:)? Or worse: (?
- Make at least three ideas for improving your model.
## TODO: On your computer, in step 6, run your algorithm on at least 6 images.
## Freely use the required number of code units
from keras.preprocessing import image
import matplotlib.pyplot as plt
From PIL import Image
import urllib.request
import io
from tqdm import tqdm
URL = [ 'https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/12233437/Entlebucher-Mountain-Dog-On-White-01.jpg' ,
'https://pre00.deviantart.net/827b/th/pre/f/2018/147/b/2/img_20180526_134216_20180526220640765_by_regnoart-dccq3ey.jpg' ,
'https://c8.alamy.com/comp/ABXTB4/ dog-wirehaired-parson-jack-russell-terrier-profile-pr-224-ABXTB4.jpg' ,
'https://ragom.org/sites/default/files/styles/dog_bio_hero_309/public/images/dogs/2015/ Twix_15-224/Twix15224b1.jpg?itok=D08C-hju' ,
'https://upload.wikimedia.org/wikipedia/commons/thumb/5/56/Donald_Trump_official_portrait.jpg/220px-Donald_Trump_official_portrait.jpg' ,
'https://d1u5p3l4wpay3k.cloudfront.net/fallout_gamepedia/thumb/9/94/Mao.jpg/251px-Mao.jpg?version=b5a173a0cc381c1db1e68c0de759f8d9' ,
'https://i.vimeocdn.com/portrait/11854029_300x300' ,
'http://p9.pstatp.com/large/1852/4456914283' ,]
Fig = plt . figure ( figsize = ( 8 , 8 ))
Row = 3
col = 4
for i , url in enumerate ( tqdm ( URL )):
with urllib . request . urlopen ( url ) as url :
f = io . BytesIO ( url . read ())
img = Image . open ( f )
img = img . resize (( 224 , 224 ), Image . ANTIALIAS )
IMG = Image . img_to_array ( IMG )
Fig . add_subplot ( Row , COL , I + . 1 )
PLT . imshow ( IMG / 255. )
X = IMG
OUT = Predict ( X )
PLT . title ( get_name ( OUT ))
PLT . Show ()
- The result is bad since the model thinks every dog is a Bull_terrier
- There are some problems of this model
- Firstly, it cannot handle adversarial inputs since our model only trained without augmentation. We can augmente data (flip, rotate, and add random noise)
- Secondly, the model may think cats are dogs because there are only dogs in our training sample. We can add more categories.
- Thirdly, the prediction need a lot of time to generate on giant network and has a potential of overfitting. We can add weight decade.
Note: When you have written all the code and answered all the questions. You can export your iPython Notebook as an HTML file. You can do this in the menu bar by exporting File -> Download as -> HTML (.html) as the job submission for this HTML and this iPython notebook.
