Understanding Deep Neural Networks Training Course

Part 1 – Deep Learning and DNN Concepts

Introduction to AI, Machine Learning & Deep Learning

• Exploring the history and core concepts of artificial intelligence, distinguishing practical applications from common misconceptions.
• Collective Intelligence: The process of aggregating knowledge shared across multiple virtual agents.
• Genetic algorithms: Techniques for evolving populations of virtual agents through selection mechanisms.
• Foundational definitions of Learning Machines.
• Task categorization: supervised learning, unsupervised learning, and reinforcement learning.
• Action types: classification, regression, clustering, density estimation, and dimensionality reduction.
• Illustrative examples of Machine Learning algorithms, including Linear regression, Naive Bayes, and Random Tree.
• Comparing Machine learning vs. Deep Learning: Identifying problems where Machine Learning remains the state-of-the-art (e.g., Random Forests & XGBoosts).

Basic Concepts of a Neural Network (Application: multi-layer perceptron)

• A refresher on essential mathematical foundations.
• Defining neural networks: classical architectures, activation functions,
• Weighting of previous activations and determining network depth.
• Defining network learning processes: cost functions, back-propagation, Stochastic gradient descent, and maximum likelihood.
• Modeling neural networks: Mapping input and output data based on problem type (regression, classification, etc.), including the curse of dimensionality.
• Differentiating between multi-feature data and signals; selecting appropriate cost functions based on data characteristics.
• Function approximation by neural networks: Theory and practical examples.
• Distribution approximation by neural networks: Theory and practical examples.
• Data Augmentation: Strategies for balancing datasets.
• Generalization of results from neural networks.
• Initialization and regularization techniques: L1 / L2 regularization and Batch Normalization.
• Optimization algorithms and convergence strategies.

Standard ML / DL Tools

This section provides a comparative overview of key tools, outlining their advantages, disadvantages, ecosystem positioning, and intended use cases.

• Data management platforms: Apache Spark and Apache Hadoop.
• Machine Learning libraries: Numpy, Scipy, and Sci-kit.
• High-level Deep Learning frameworks: PyTorch, Keras, and Lasagne.
• Low-level Deep Learning frameworks: Theano, Torch, Caffe, and Tensorflow.

Convolutional Neural Networks (CNN).

• Overview of CNNs: Fundamental principles and primary applications.
• Core operations: Convolutional layers and the use of kernels.
• Techniques such as Padding & stride, feature map generation, and pooling layers; including 1D, 2D, and 3D extensions.
• Review of CNN architectures that have established the state of the art in classification.
• Architectural innovations: LeNet, VGG Networks, Network in Network, Inception, and Resnet, including applications of 1x1 convolutions and residual connections.
• Implementation of attention models.
• Practical application to common classification tasks (text or image).
• CNNs for generative tasks: Super-resolution and pixel-to-pixel segmentation.
• Key strategies for enhancing feature maps in image generation.

Recurrent Neural Networks (RNN).

• Overview of RNNs: Core principles and practical applications.
• Fundamental operations: Hidden activations, back propagation through time, and unfolded representations.
• Advancements towards Gated Recurrent Units (GRUs) and LSTM (Long Short Term Memory).
• Evolution of architectures and their respective improvements.
• Addressing convergence challenges and the vanishing gradient problem.
• Standard architectures: Time series prediction and classification tasks.
• RNN Encoder-Decoder architectures and the integration of attention models.
• NLP applications: Word / character encoding and machine translation.
• Video applications: Predicting the next frame in a video sequence.

Generative models: Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN).

• Introduction to generative models and their relationship with CNNs.
• Auto-encoders: Dimensionality reduction and constrained generation.
• Variational Auto-encoders: Modeling distributions, defining latent spaces, utilizing the reparameterization trick, and understanding applications and limitations.
• Generative Adversarial Networks: Core fundamentals.
• Dual Network Architecture (Generator and Discriminator), alternating learning, and available cost functions.
• GAN convergence dynamics and common implementation difficulties.
• Enhanced convergence techniques: Wasserstein GAN, Began, and Earth Moving Distance.
• Applications in image/photo generation, text generation, and super-resolution.

Deep Reinforcement Learning.

• Overview of reinforcement learning: Controlling agents within defined environments.
• Defining states and possible actions.
• Utilizing neural networks to approximate state functions.
• Deep Q Learning: Experience replay and application to video game control.
• Policy optimization: On-policy & off-policy methods, Actor-critic architecture, and A3C.
• Applications: Controlling individual video games or complex digital systems.

Part 2 – Theano for Deep Learning

Theano Basics

• Introduction to the framework.
• Installation and configuration processes.

TheanoFunctions

• Managing inputs, outputs, updates, and givens.

Training and Optimization of a neural network using Theano

• Modeling neural network structures.
• Implementing Logistic Regression.
• Integrating Hidden Layers.
• Network training procedures.
• Computation and classification tasks.
• Optimization strategies.
• Calculating Log Loss.

Testing the model

Part 3 – DNN using Tensorflow

TensorFlow Basics

• Creating, initializing, saving, and restoring TensorFlow variables.
• Feeding, reading, and preloading TensorFlow data.
• Leveraging TensorFlow infrastructure for large-scale model training.
• Visualizing and evaluating models using TensorBoard.

TensorFlow Mechanics

• Data preparation strategies.
• Downloading necessary resources.
• Defining Inputs and Placeholders.
• Building Graphs:
  • Inference operations.
  • Loss calculation.
  • Training steps.
• Training the Model:
  • Graph structure.
  • Session management.
  • Training loops.
• Evaluating the Model:
  • Constructing the Evaluation Graph.
  • Interpreting Eval Output.

The Perceptron

• Selection and use of activation functions.
• The perceptron learning algorithm.
• Binary classification using perceptrons.
• Document classification with perceptrons.
• Inherent limitations of the perceptron model.

From the Perceptron to Support Vector Machines

• Understanding Kernels and the kernel trick.
• Maximum margin classification and identifying support vectors.

Artificial Neural Networks

• Handling Nonlinear decision boundaries.
• Feedforward vs. feedback neural network architectures.
• Multilayer perceptrons.
• Minimizing cost functions effectively.
• Forward propagation mechanics.
• Back propagation mechanics.
• Strategies to improve neural network learning efficiency.

Convolutional Neural Networks

• Defining project goals.
• Designing the Model Architecture.
• Underlying principles of operation.
• Structuring code for maintainability.
• Launching and Training the Model.
• Evaluating model performance.

Basic Introductions to the following modules (Brief overviews provided subject to time availability):

Tensorflow - Advanced Usage

• Implementing Threading and Queues.
• Setting up Distributed TensorFlow.
• Writing documentation and sharing models.
• Customizing Data Readers.
• Manipulating TensorFlow model files.

TensorFlow Serving

• Introduction to serving concepts.
• Basic Serving Tutorial.
• Advanced Serving Tutorial.
• Serving the Inception Model Tutorial.