Training Session 1: Project Scaffolding Standards for AI/ML Projects¶

Objective¶

By the end of this session, participants will understand the importance of standardized project structures, their advantages, and how to adapt directory structures for different project types (e.g., Python projects, Gen AI applications, production-ready ML pipelines).

1. Introduction: The Importance of Project Scaffolding Standards¶

Why Standardized Project Structures Matter¶

In AI/ML projects, we work in a multidisciplinary environment that often includes:

• Data Scientists: Focused on creating models and analyzing data.
• ML Engineers: Responsible for scaling and deploying models.
• Data Engineers: Handling data pipelines and storage.

Goals of Project Scaffolding¶

1. Organization
   Avoid chaos by keeping files and folders structured, making navigation intuitive.

2. Collaboration
   Ensure team members can work on the same project without confusion about file locations or naming conventions.

3. Reproducibility
   Guarantee that others (or even your future self!) can reproduce experiments and workflows seamlessly.

4. Scalability
   Design projects that can grow, transitioning from proof-of-concept to production without major restructuring.

Common Challenges Without Standardization¶

Without a standardized project scaffold:

• Onboarding New Team Members:
  • Takes longer, as they need to learn the structure of every new project.
• Debugging and Maintenance:
  • Becomes harder as team members struggle to locate key files or understand dependencies.
• Collaboration Friction:
  • Mismatched expectations around file organization slow down progress.
• Code Reusability:
  • Is hampered, as inconsistent structures make it harder to integrate components across projects.

What We’ll Cover Today¶

1. The Importance of Standardization: Why a structured approach is critical for team projects.
2. Advantages of a Standardized Structure: Benefits for organization, collaboration, reproducibility, and scalability.
3. Types of Directory Structures: Differences between Python utilities, Gen AI applications, and production ML pipelines.
4. Exploring Our Standard: Walkthrough of the directory structure we’ll use as a template.
5. Interactive Activity: Hands-on exercise to restructure a disorganized project.

Key Takeaways¶

• Standardized project scaffolding is the foundation for professional, collaborative AI/ML development.
• It enhances productivity, reduces confusion, and sets your project up for long-term success.
• The skills you’ll learn today will improve how you work within teams and across roles.

2. The Importance of Standardized Directory Structures¶

1. Collaboration¶

When multiple team members—data scientists, ML engineers, and data engineers—work on the same project, clarity is key. A standardized directory structure promotes seamless collaboration by:

• Providing Clear File Locations:
  • Everyone knows where to find code, data, configurations, and documentation.
  • Reduces time wasted searching for files or asking teammates for clarification.

• Minimizing Miscommunication:
  • Clear separation of concerns (e.g., raw vs. processed data, scripts vs. tests) eliminates confusion over file ownership or purpose.
  • Agreed-upon standards reduce the risk of overwriting each other's work.

• Enabling Parallel Development:
  • Teams can work on different aspects of the project (e.g., data preprocessing, model training, and deployment) without stepping on each other's toes.

2. Reproducibility¶

Reproducibility is a cornerstone of AI/ML projects, especially when experiments need to be validated or results are handed off to other teams. Standardization supports reproducibility by:

• Ensuring Consistent Experiment Replication:
  • Clearly organized data/ directories (e.g., raw, processed) ensure that the same inputs produce the same outputs.
  • Version-controlled configuration files (settings.yaml, config.cfg) lock down parameters for experiments.

• Providing a Clear Workflow:

  • Scripts are logically separated (e.g., scripts/preprocess_data.py, scripts/train_model.py), ensuring that the process is documented and repeatable.

• Facilitating Debugging:

  • When something breaks, team members can easily trace issues back to specific scripts or data folders.

3. Scalability¶

Projects often start as small proofs of concept but need to evolve into production-ready systems. A standardized structure allows your project to scale by:

• Future-Proofing for Growth:
  • Clear separation of modules (e.g., src/, tests/, docs/) ensures the project can handle new features without becoming unmanageable.

• Simplifying Transitions to Production:

  • Production-ready components like Dockerfile, deployment scripts, and CI/CD workflows can be easily integrated without restructuring the entire project.

• Accommodating Team Expansion:

  • A consistent structure ensures new team members can quickly onboard and contribute without disrupting existing workflows.

Practical Examples¶

1. Collaboration:
   • Imagine two data scientists working on feature engineering and model training. If both use different folder names or scatter files across the project, merging their work becomes chaotic. A standardized structure ensures they can work independently while aligning with the broader project.

1. Reproducibility:
   • A research project involving multiple experiments benefits greatly from a clear config/ directory where all experiment parameters are stored. Another team can pick up the same configuration and rerun the experiments without confusion.

1. Scalability:
   • A POC chatbot developed in a single main.py script can grow into a production system with modules for retrievers/, generators/, and evaluation/. Standardization ensures this evolution is smooth.

Key Takeaways¶

• Collaboration: Reduces confusion and improves teamwork by providing clarity.
• Reproducibility: Ensures that experiments and workflows can be repeated and validated.
• Scalability: Prepares the project for future growth, whether it’s transitioning to production or adding team members.

Next Up: Let’s explore how different types of projects (e.g., Python scripts, Gen AI applications, production ML pipelines) require variations in directory structures.

3. Common Pitfalls of Poor Project Structures¶

1. Lack of Reproducibility¶

Reproducibility is essential in AI/ML projects, especially when results need to be validated, shared, or reproduced at a later date. Without a clear structure:

• Experiment Replication Becomes Impossible:
  • Raw data might be overwritten or mixed with processed data, leading to inconsistent results.
  • Missing or hardcoded configurations make it difficult to replicate an experiment.

• Code and Data Mismatches:
  • Scripts might be scattered across folders without clear versioning or alignment with datasets.
  • Dependencies might not be documented, causing issues when trying to rerun old workflows.

Example:

2. Increased Debugging Time¶

When projects lack clear organization, debugging becomes a frustrating and time-consuming process:

• Difficult to Locate Issues:
  • Without a dedicated logs/ folder or clear script separation, tracing errors to their origin takes significantly longer.

• Unclear Code Ownership:
  • If scripts are dumped in a single folder, it’s challenging to determine which script is responsible for what functionality.

• Duplication and Confusion:
  • Duplicate or outdated scripts might exist, leading to confusion about which file to debug.

Example:

3. Difficulties Onboarding Team Members¶

A poorly structured project can discourage new team members and significantly delay their productivity:

• Longer Ramp-Up Time:
  • New contributors spend excessive time trying to understand where things are located and how they work.

• Knowledge Silos:
  • If only one person understands the project layout, they become a bottleneck for progress.

• Higher Error Risk:
  • New team members might inadvertently disrupt workflows by modifying the wrong files or folders.

Example:

Key Takeaways¶

• Lack of reproducibility undermines trust in results and creates unnecessary rework.
• Increased debugging time leads to wasted effort and frustration.
• Difficulties onboarding new members slow down team productivity and create bottlenecks.

Next Up: Let’s explore how different types of AI/ML projects require tailored directory structures and how our standardized approach addresses these pitfalls.

4. Exploring Different Directory Structures¶

4.1 Python Project Structure for Simple Scripts and Utilities¶

When to Use This Structure¶

This structure is ideal for: - Small projects, such as data manipulation utilities or command-line tools. - Projects involving datasets, even small ones, that require preprocessing or feature engineering. - Initial exploration and experimentation using Jupyter notebooks. - Prototypes or proofs of concept before scaling into larger applications.

Recommended Directory Structure¶

Here’s the recommended directory structure for simple projects, including a data/ section:

project-name/
├── project_name/                 <- Source code for the project.
│   ├── __init__.py               <- Marks the directory as a Python package.
│   ├── main.py                   <- Main script or entry point of the project.
│   └── utils.py                  <- Helper functions used by `main.py`.
├── data/                         <- Data used in the project.
│   ├── raw                       <- Original, immutable data dump.
│   ├── external                  <- Data from third-party sources.
│   ├── interim                   <- Intermediate data, partially processed.
│   ├── processed                 <- Fully processed data, ready for analysis.
│   └── features                  <- Engineered features ready for model training.
├── notebooks/                    <- Jupyter notebooks for exploration and experimentation.
│   ├── data_exploration.ipynb    <- Notebook for data exploration.
│   ├── prototyping.ipynb         <- Notebook for prototyping and initial analysis.
│   └── README.md                 <- Guidelines for using the notebooks.
├── tests/                        <- Unit and integration tests.
│   ├── __init__.py               <- Marks the directory as a package for testing.
│   ├── test_main.py              <- Tests for `main.py`.
│   └── test_utils.py             <- Tests for `utils.py`.
├── .gitignore                    <- Files and directories to ignore in Git.
├── pyproject.toml                <- Project configuration for dependencies and tools.
├── README.md                     <- Description and instructions for the project.
└── LICENSE                       <- License file for open-source projects.

Detailed Explanation¶

1. Data Directory (data/)¶

This directory organizes all data used in the project, ensuring a clear workflow from raw data to processed features. Standard subdirectories include:

• raw/:

  • Contains the original data that is immutable and serves as the source of truth.
  • Example: .csv or .json files downloaded from a database or external source.

• external/:

  • Holds data from third-party sources, such as APIs or shared datasets.
  • Example: Pre-trained embeddings, external .zip files.

• interim/:

  • Stores intermediate data that has been partially processed.
  • Example: Data after initial cleaning or aggregation but before full preprocessing.

• processed/:

  • Contains fully processed data that is ready for analysis or modeling.
  • Example: Cleaned and structured data in .parquet or .csv formats.

• features/:

  • Holds engineered features for model training.
  • Example: Feature matrices in .npy or .pkl formats.

2. Integration with notebooks/¶

The data/ directory complements the notebooks/ directory: - Use data/raw/ for initial exploration in data_exploration.ipynb. - Save interim results in data/interim/ for reuse across notebooks and scripts. - Generate final datasets in data/processed/ and data/features/ for downstream tasks like modeling.

3. Integration with Source Code (project_name/)¶

The data/ directory interacts directly with source code: - utils.py: Functions for loading and saving data from specific directories. - main.py: Accesses processed data or features for the core functionality.

Example utility functions:

import os
import pandas as pd

def load_raw_data(file_name: str) -> pd.DataFrame:
    """Loads a raw data file from the `data/raw/` directory."""
    file_path = os.path.join("data", "raw", file_name)
    return pd.read_csv(file_path)

def save_processed_data(df: pd.DataFrame, file_name: str):
    """Saves a processed DataFrame to the `data/processed/` directory."""
    file_path = os.path.join("data", "processed", file_name)
    df.to_csv(file_path, index=False)

Examples of Projects with data/ and notebooks/¶

1. Data Cleaning Tool¶

Use Case: Clean and preprocess raw customer data, with notebooks for exploration and a structured data workflow.

Directory structure:

data-cleaner/
├── data/
│   ├── raw/
│   ├── external/
│   ├── interim/
│   ├── processed/
│   └── features/
├── notebooks/
│   ├── data_exploration.ipynb
│   └── prototyping.ipynb
├── data_cleaner/
│   ├── __init__.py
│   ├── main.py
│   └── cleaning_utils.py

2. Feature Engineering Project¶

Use Case: Engineer features for predictive modeling, with raw data, interim transformations, and a focus on creating reusable features.

Directory structure:

feature-engineer/
├── data/
│   ├── raw/
│   ├── interim/
│   ├── processed/
│   └── features/
├── notebooks/
│   ├── feature_exploration.ipynb
│   └── prototyping.ipynb
├── feature_engineer/
│   ├── __init__.py
│   ├── main.py
│   └── feature_utils.py

Key Takeaways¶

• The data/ directory introduces a clear workflow for handling raw, interim, processed, and engineered datasets.
• Combined with notebooks/, this structure supports iterative experimentation and scalable development.
• Even simple projects benefit from modular data organization, improving reproducibility and collaboration.

Next Up: Let’s explore how directory structures adapt for Gen AI applications and production-ready ML pipelines.

4.2 Exploring Standards for Organizing Python Source Code¶

1. Flat Directory Approach¶

Overview¶

The source code is located directly under the top-level directory, inside a folder named after the project.

project-name/
├── project_name/                 <- Source code for the project.
│   ├── __init__.py               <- Marks the directory as a Python package.
│   ├── main.py                   <- Main script or entry point of the project.
│   └── utils.py                  <- Helper functions used by `main.py`.

Advantages¶

• Simplicity:
  • Easier to navigate and set up for small projects.
  • Ideal for beginners and small utility projects.

• No Additional Layers:
  • Directly accessible from the project root, making it easy to run scripts or modules without configuring paths.
• Intuitive for Small Projects:
  • Reduces overhead when the project scope is limited, such as single-purpose utilities.

Disadvantages¶

• Risk of Name Collisions:
  • The top-level package shares its name with the project directory, leading to potential import issues (e.g., importing project_name may conflict with the folder name).

• Scaling Limitations:
  • Mixing source code with top-level directories like data/, notebooks/, and tests/ can become unwieldy as the project grows.

When to Use¶

• Best for:
  • Small, single-purpose projects that are unlikely to expand significantly.
  • Scripts or utilities where simplicity is a priority.

2. Nested src/ Directory Approach¶

Overview¶

The source code is placed inside a src/ directory, which contains the main package folder.

project-name/
├── src/project_name/             <- Source code for the project.
│   ├── __init__.py               <- Marks the directory as a Python package.
│   ├── main.py                   <- Main script or entry point of the project.
│   └── utils.py                  <- Helper functions used by `main.py`.

Advantages¶

• Clear Separation:
  • Distinguishes source code from other directories, such as data/, tests/, and notebooks/, ensuring a cleaner and more professional project structure.
• Avoids Import Conflicts:
  • Prevents Python from accidentally importing modules from the top-level directory, reducing the risk of name collisions.
• Better for Larger Projects:
  • Encourages scalability and organization when the project involves multiple packages or modules.

Disadvantages¶

• Added Complexity:
  • Requires configuring the Python path to include src/ (e.g., setting PYTHONPATH or using tools like Poetry or pytest to handle paths).

• Overhead for Small Projects:
  • The additional layer might feel unnecessary for simple utilities or one-off scripts.

When to Use¶

• Best for:
  • Medium to large-scale projects where clean separation of source code is critical.
  • Projects intended for production or distribution as Python packages.

Comparison of the Two Approaches¶

Recommendations¶

Flat Directory Approach:¶

• Best for:
  • Small projects with a narrow focus (e.g., data cleaning tools, simple CLI utilities).
  • Prototyping or proof-of-concept work.
• Avoid if:
  • The project is expected to grow significantly.
  • The name of the package risks conflicting with the project directory.

Nested src/ Directory Approach:¶

• Best for:
  • Production-level projects involving multiple components or packages.
  • Projects requiring clean, scalable structures.
• Avoid if:
  • The project scope is very small, and the extra complexity is unnecessary.

Key Takeaways¶

4.3 Production-Ready ML Project Structure¶

Use Cases¶

Production-ready ML project structures are ideal for:

• End-to-End Machine Learning Pipelines:
  • Managing data ingestion, preprocessing, training, evaluation, and deployment.
• Collaboration Between Teams:
  • Facilitating clear roles and responsibilities in teams with multiple contributors.
• Scalable and Deployable Solutions:
  • Moving from experimental prototypes to fully operationalized systems in production.

Key Features of the Structure¶

1. Clear Separation of Data¶

The data/ directory enables reproducibility and clarity through organized subdirectories:

• data/raw/: Contains unprocessed, immutable data dumps.
• data/external/: Stores third-party or external datasets.
• data/interim/: Holds intermediate results during preprocessing.
• data/processed/: Contains clean datasets ready for modeling.
• data/features/: Includes feature matrices generated during preprocessing.

2. Modular Scripts¶

Scripts are divided based on functionality, promoting a clean and reusable workflow:

• scripts/preprocess_data.py: Handles data cleaning, transformation, and feature engineering.
• scripts/train_model.py: Contains the logic for model training.
• scripts/evaluate_model.py: Includes evaluation metrics and model performance validation.
• scripts/deploy_model.py: Manages deployment of the trained model to production.

3. Robust Testing Framework¶

A dedicated tests/ directory ensures that every component of the pipeline is thoroughly validated:

• Unit Tests:
  • Validate individual functions, such as data preprocessing or model training utilities.
• Integration Tests:
  • Test how different pipeline components interact with each other.
• End-to-End Tests:
  • Simulate the full pipeline from raw data ingestion to model deployment.

4. CI/CD Workflows¶

Continuous Integration and Continuous Deployment (CI/CD) workflows automate quality checks and streamline deployment:

• GitHub Actions:
  • Automate testing, linting, and formatting using tools like Black, Ruff, and Pytest.
• Docker:
  • Containerize the pipeline for consistent execution across environments.
• Deployment Pipelines:
  • Automate deployment to production environments using CI/CD tools like Jenkins or GitHub Actions.

name: ML Pipeline CI/CD

on:
  push:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.8"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install poetry
          poetry install
      - name: Run tests
        run: poetry run pytest
  deploy:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v3
      - name: Deploy with Docker
        run: docker build -t my-ml-project .

5. Inclusion of a Dockerfile¶

The Dockerfile ensures consistent execution across environments by containerizing the project:

# Dockerfile for Production-Ready ML Project
FROM python:3.8-slim

# Set up working directory
WORKDIR /app

# Install dependencies
COPY pyproject.toml poetry.lock ./
RUN pip install poetry && poetry install --no-dev

# Copy source code
COPY src/ /app/src/

# Entry point
CMD ["python", "src/main.py"]

Example Directory Structure¶

project-name/
├── data/
│   ├── raw/                       # Original, immutable data dump
│   ├── external/                  # Data from third-party sources
│   ├── interim/                   # Intermediate, partially processed data
│   ├── processed/                 # Fully processed data
│   └── features/                  # Engineered features for modeling
├── scripts/
│   ├── preprocess_data.py         # Data cleaning and transformation
│   ├── train_model.py             # Model training logic
│   ├── evaluate_model.py          # Evaluation metrics and validation
│   ├── deploy_model.py            # Deployment script
├── src/
│   ├── __init__.py                # Marks the directory as a Python package
│   ├── data/                      # Data loading and preprocessing
│   ├── models/                    # Model definitions and utilities
│   ├── evaluation/                # Evaluation metrics and methods
│   ├── utils.py                   # General utility functions
│   └── main.py                    # Entry point for the application
├── tests/
│   ├── __init__.py                # Marks the directory as a Python package
│   ├── test_preprocessing.py      # Unit tests for preprocessing
│   ├── test_training.py           # Unit tests for model training
│   └── test_end_to_end.py         # End-to-end pipeline tests
├── Dockerfile                     # Dockerfile for containerizing the project
├── pyproject.toml                 # Dependency management and configuration
├── poetry.lock                    # Lock file for dependencies
├── .github/
│   ├── workflows/
│   │   ├── test.yaml              # CI workflow for testing
│   │   ├── lint.yaml              # CI workflow for linting and formatting
│   │   └── deploy.yaml            # CI workflow for deployment
├── README.md                      # Overview and usage instructions
└── LICENSE                        # License for the project

Key Takeaways¶

This structure is the gold standard for end-to-end machine learning pipelines, ensuring scalability, maintainability, and production readiness.

4.4. Creating Python Packages for Specialized Fine-Tuning and Data Inclusion¶

When to Create a Python Package¶

Python packages are beneficial when:

1. Avoiding Overcrowding:
   • Separating fine-tuning workflows or specialized functionality keeps the main project clean.
2. Reusability Across Projects:
   • A fine-tuning package can be easily imported into multiple Gen AI or ML projects.
3. Distributing Data or Models:
   • Including datasets, trained models, or configuration files ensures accessibility and version control.

Recommended Directory Structure for a Python Package with Data¶

Here’s a directory structure for a Python package designed for fine-tuning models and embedding data:

fine_tune_package/
├── src/
│   └── fine_tune_package/
│       ├── __init__.py              # Marks the directory as a Python package.
│       ├── datasets.py              # Code for accessing included data.
│       ├── fine_tune.py             # Core functionality for fine-tuning.
│       ├── utils.py                 # Helper functions for the package.
│       ├── data/                    # Directory containing embedded data.
│       │   ├── __init__.py          # Marks the directory as a subpackage.
│       │   └── model_config.yaml    # Example model configuration file.
│       │   └── vocab.txt            # Vocabulary or tokenizer data.
│       └── models/                  # Directory for pre-trained or fine-tuned models.
│           ├── __init__.py          # Marks the directory as a subpackage.
│           └── fine_tuned_model.bin # Binary file of the fine-tuned model.
├── tests/
│   ├── __init__.py                  # Marks the directory as a package for tests.
│   ├── test_fine_tune.py            # Tests for fine-tuning functionality.
│   └── test_datasets.py             # Tests for dataset access.
├── pyproject.toml                   # Configuration for package building and dependencies.
├── README.md                        # Package description and usage examples.
└── LICENSE                          # License for the package.

Including Data in a Package¶

Use Cases¶

1. Required for Functionality:
   • Data files like tokenizers or model configurations are essential for package operation.
2. Example Data:
   • Including sample datasets to demonstrate package functionality.
3. Reproducibility:
   • Bundling data ensures code and data are synchronized and version-controlled.

Embedding Data Using importlib.resources¶

Step 1: Add Data to the Package¶

Place required data files (e.g., model_config.yaml) in the data/ subpackage.

fine_tune_package/
├── src/
│   └── fine_tune_package/
│       ├── data/
│       │   ├── __init__.py
│       │   └── model_config.yaml

Step 2: Create a Helper Function to Access the Data¶

Use importlib.resources to access embedded files.

from importlib import resources
import yaml

def get_model_config():
    """Get the model configuration file as a dictionary."""
    with resources.path("fine_tune_package.data", "model_config.yaml") as f:
        with open(f, "r") as file:
            config = yaml.safe_load(file)
    return config

Step 3: Access the Data in Your Code¶

Use the helper function to access the configuration file.

from fine_tune_package.datasets import get_model_config

config = get_model_config()
print(config)

Downloading Large Data Files¶

For large datasets or models, include scripts for dynamic downloading.

import os
import requests

def download_model(destination="models/fine_tuned_model.bin"):
    """Download a fine-tuned model file from an external source."""
    url = "https://example.com/fine_tuned_model.bin"
    os.makedirs(os.path.dirname(destination), exist_ok=True)
    response = requests.get(url, stream=True)
    with open(destination, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)
    print(f"Model downloaded to {destination}")

Building and Installing the Package¶

Step 1: Update pyproject.toml¶

Configure the package for the src layout and specify dependencies.

[tool.poetry]
name = "fine-tune-package"
version = "0.1.0"
description = "A package for fine-tuning Gen AI models."
authors = ["Your Name <your.email@example.com>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28"
pyyaml = "^6.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Step 2: Install the Package Locally¶

Install the package in editable mode for development:

poetry install

Key Benefits of This Approach¶

1. Reusability:
   • Install and reuse the package across multiple projects.
2. Data Accessibility:
   • Embedded data or download scripts ensure smooth workflows.
3. Scalability:
   • Modular packages reduce code clutter and improve maintainability.

Key Takeaways¶

5. Overview of the Standard Directory Structure¶

├── .devcontainer                  <- Directory for Visual Studio Code Dev Container configuration.
│   └── devcontainer.json          <- Configuration file for defining the development container.
├── .github                        <- Directory for GitHub-specific configuration and metadata.
│   ├── CODEOWNERS                 <- File to define code owners for the repository.
│   ├── CONTRIBUTING.md            <- Guidelines for contributing to the project.
│   └── pull_request_template.md   <- Template for pull requests to standardize and improve PR quality.
├── .vscode                        <- Directory for Visual Studio Code-specific configuration files.
│   ├── cspell.json                <- Configuration file for the Code Spell Checker extension.
│   ├── dictionaries               <- Directory for custom dictionary files.
│   │   └── data-science-en.txt    <- Custom dictionary for data science terminology.
│   ├── extensions.json            <- Recommended extensions for the project.
│   └── settings.json              <- Workspace-specific settings for Visual Studio Code.
├── config                         <- Configuration files for the project.
├── data                           <- Data for the project, divided into different stages of data processing.
│   ├── raw                        <- Original, immutable data dump.
│   ├── external                   <- Data from third-party sources.
│   ├── interim                    <- Intermediate data, partially processed.
│   ├── processed                  <- Fully processed data, ready for analysis.
│   └── features                   <- Engineered features ready for model training.
├── docs                           <- Documentation for the project.
│   ├── api-reference.md           <- API reference documentation.
│   ├── explanation.md             <- Detailed explanations and conceptual documentation.
│   ├── how-to-guides.md           <- Step-by-step guides on performing common tasks.
│   ├── index.md                   <- The main documentation index page.
│   └── tutorials.md               <- Tutorials related to the project.
├── log                            <- Logs generated by the project.
├── models                         <- Machine learning models, scripts, and other related artifacts.
├── notebooks                      <- Jupyter notebooks for experiments, examples, or data analysis.
├── scripts                        <- Directory for project-specific scripts and utilities.
│   └── hooks                      <- Directory for custom git hooks and other automation scripts.
│       ├── branch-name-check.sh   <- Hook script for checking branch names.
│       ├── commit-msg-check.sh    <- Hook script for checking commit messages.
│       ├── filename-check.sh      <- Hook script for checking file names.
│       ├── generate_docs.sh       <- Script for generating documentation.
│       └── restricted-file-check.sh <- Hook script for checking restricted files.
├── src                            <- Source code for the project.
│   └── collaborativeaitoy         <- Main project module.
│       ├── __init__.py            <- Initializes the Python package.
│       ├── main.py                <- Entry point for the application.
│       ├── app.py                 <- Main application logic.
│       └── utils.py               <- Utility functions.
├── tests                          <- Directory for all project tests.
│   ├── integration                <- Integration tests.
│   └── spec                       <- Specification tests (unit tests).
├── .gitignore                     <- Specifies intentionally untracked files to ignore.
├── .pre-commit-config.yaml        <- Configuration for pre-commit hooks.
├── Dockerfile                     <- Dockerfile for containerizing the application.
├── Makefile                       <- Makefile with commands like `make data` or `make train`.
├── mkdocs.yml                     <- Configuration file for MkDocs, a static site generator for project documentation.
├── pyproject.toml                 <- Configuration file for Python projects which includes dependencies and package information.
├── README.md                      <- The top-level README for developers using this project.
└── .env                           <- Environment variables configuration file (not visible).

1. .devcontainer¶

2. .github¶

3. .vscode¶

4. config¶

5. data¶

6. docs¶

7. log¶

8. models¶

9. notebooks¶

10. scripts¶

11. src¶

12. tests¶

13. Key Root-Level Files¶

Conclusion¶

6. Automating Standards with Cookiecutter¶

What is Cookiecutter?¶

Why Use Cookiecutter?¶

Step-by-Step Demonstration¶

Step 1: Install Cookiecutter¶

pip install cookiecutter

Step 2: Choose or Create a Template¶

Example Template: ML Project Structure¶

cookiecutter-ml-template/
├── {{cookiecutter.project_slug}}/
│   ├── data/
│   │   ├── raw/
│   │   ├── processed/
│   ├── notebooks/
│   │   └── example_notebook.ipynb
│   ├── src/
│   │   ├── __init__.py
│   │   ├── main.py
│   │   └── utils.py
│   ├── tests/
│   │   ├── __init__.py
│   │   └── test_main.py
│   ├── .gitignore
│   ├── README.md
│   ├── pyproject.toml
│   └── LICENSE
├── cookiecutter.json
└── README.md

Step 3: Configure the Template¶

Define prompts and their default values in cookiecutter.json:

{
    "project_name": "My ML Project",
    "project_slug": "{{ cookiecutter.project_name.lower().replace(' ', '_') }}",
    "author_name": "Your Name",
    "description": "A machine learning project template.",
    "license": "MIT"
}

Step 4: Generate a New Project¶

Run Cookiecutter with the template:

cookiecutter https://github.com/your-org/cookiecutter-ml-template

Example input prompts:

project_name [My ML Project]: Awesome AI Project
author_name [Your Name]: Jane Doe
description [A machine learning project template.]: Fine-tuning Gen AI Models
license [MIT]: Apache-2.0

Generated structure:

awesome_ai_project/
├── data/
├── notebooks/
├── src/
├── tests/
├── README.md
├── pyproject.toml
├── LICENSE
└── .gitignore

Step 5: Customize and Use the Project¶

Navigate to the generated directory:

cd awesome_ai_project
poetry install

Advanced Features¶

Hooks¶

Private Templates¶

Store templates in private GitHub repositories to maintain confidentiality and customize for internal use.

Integration with CI/CD¶

Automate project generation as part of a CI/CD pipeline to maintain consistency across deployments.

Key Takeaways¶

• Automation and Standardization: Cookiecutter ensures consistent and professional project setups.
• Scalability: Templates can evolve with team standards and project complexity.
• Reusability: Save time and reduce errors by leveraging pre-built templates.

Next Steps: Explore existing templates or customize one for your team’s ML or Gen AI projects.

7. Advanced Topics and Future Considerations¶

How the Structure Supports CI/CD Workflows¶

1. Automated Testing and Quality Assurance¶

Example Workflow Configuration:

name: CI Workflow

on:
  push:
    branches:
      - main

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      - name: Install Dependencies
        run: |
          pip install poetry
          poetry install
      - name: Run Tests
        run: poetry run pytest

2. Deployment Pipelines¶

Transitioning from POCs to Production¶

1. Modular Design¶

2. Collaboration Readiness¶

Documentation in docs/ and README.md enables seamless handoffs between teams and stakeholders.

Documentation Tools: MkDocs¶

What is MkDocs?¶

MkDocs converts Markdown files (e.g., in docs/) into professional static documentation sites.

Benefits of MkDocs¶

• Centralized Documentation: Consolidates all guides and references.
• Ease of Use: Markdown-based editing simplifies updates.
• Live Preview: View changes locally during development.

Key Takeaways¶

• Automate testing, deployment, and documentation workflows for robust CI/CD integration.
• Modularize code and document comprehensively for a seamless transition from POCs to production.
• Leverage MkDocs for high-quality, accessible project documentation.

Next Steps: Implement CI/CD pipelines and explore MkDocs for your project documentation.