Traditional software begins with rules written by a programmer. Machine learning begins when those rules become too numerous, ambiguous, or unstable to write manually.

Suppose a support system must route tickets into categories such as:

Billing
Account access
Technical issue
Cancellation

A rule-based implementation might begin like this:

def classify_ticket(text: str) -> str:
    text = text.lower()

    if "refund" in text or "charged" in text:
        return "billing"

    if "password" in text or "login" in text:
        return "account_access"

    if "cancel" in text:
        return "cancellation"

    return "technical_issue"

This works until users write:

“Why did money leave my card twice?”
“The application no longer recognizes my credentials.”
“Please stop renewing my plan next month.”
“I can sign in, but the dashboard never loads.”

The programmer could keep adding conditions, but natural language has too many valid variations. Rules overlap, exceptions accumulate, and changing one condition may break another.

Machine learning offers a different approach:

Instead of explicitly programming every decision rule, provide examples and use an algorithm to discover a function that predicts the desired output.

The result is still software, but part of its behaviour now comes from data rather than source code alone.

1. Code versus a learned model

Ordinary software can be represented as:

[ \text{Input} + \text{Rules} \rightarrow \text{Output} ]

Machine-learning training looks more like:

[ \text{Inputs} + \text{Expected outputs} \rightarrow \text{Learned rules} ]

After training, inference becomes:

[ \text{New input} + \text{Learned rules} \rightarrow \text{Predicted output} ]

A learned model is therefore a parameterized function:

[ \hat{y} = f(x; \theta) ]

Where:

(x) is the input.
(\hat{y}) is the prediction.
(\theta) represents parameters learned from data.

In a normal function, a developer writes the logic. In a learned model, a developer chooses the model family, data, features, training process, and evaluation criteria. Training determines the specific parameter values.

This does not remove engineering. It moves some of the specification from code into:

The dataset
The labels
The objective function
The evaluation metrics
The deployment threshold

A model may behave incorrectly even when its training code contains no conventional bug. Its dataset may be misleading, its labels inconsistent, or its evaluation disconnected from production needs.

2. Learning requires examples

Consider a support ticket:

I was charged twice for my subscription.

The raw sentence must be represented in a form the model can process. The measurable properties supplied to a model are called features.

For text, features might include:

Whether particular words appear
Word or phrase frequencies
Ticket length
Customer plan
Product area
Language
An embedding representing semantic meaning

The expected answer is the label:

billing

Together, the input features and label form a training example:

[ (x_i, y_i) ]

A training dataset contains many such examples:

("I was charged twice", "billing")
("I cannot reset my password", "account_access")
("The mobile app keeps crashing", "technical_issue")
("Please end my subscription", "cancellation")

The model attempts to find patterns connecting inputs to labels.

The label is not automatically the truth. It is a recorded decision made by a person, system, or process. If support agents categorize similar tickets differently, the model receives contradictory supervision.

Dataset quality therefore includes:

Correctness
Consistency
Coverage
Representativeness
Freshness
Absence of unwanted shortcuts

A sophisticated algorithm cannot recover information that the dataset does not contain.

3. Different learning setups

Supervised learning

In supervised learning, every training example includes an expected output.

Examples include:

Ticket text → ticket category
Transaction → fraudulent or legitimate
Customer details → predicted resolution time
Query-document pair → relevance score

The model learns by comparing its predictions with known labels.

Unsupervised learning

In unsupervised learning, examples do not have target labels. The system attempts to discover useful structure in the inputs.

Examples include:

Grouping similar tickets
Finding unusual requests
Discovering common complaint themes
Compressing data into useful representations

Clustering ticket embeddings can reveal that many supposedly “technical” tickets concern a new payment-page failure. The algorithm can expose the group, but a person must still interpret what that group means.

Classification

Classification predicts one of several discrete categories:

billing
account_access
technical_issue
cancellation

Some problems allow exactly one category. Others are multilabel: one ticket could be both billing and account_access.

Regression

Regression predicts a numerical value, such as:

Expected resolution time
Customer satisfaction score
Estimated support cost

A regression prediction is normally continuous rather than a named category.

Ranking

Ranking orders candidates by usefulness or relevance.

A retrieval system, for example, ranks documents for a query. It is not enough to predict that several documents are relevant; the most useful ones should appear first.

Ranking is central to:

Search
Recommendation
RAG retrieval
Reranking
Tool selection

Clustering

Clustering places similar examples into groups without predefined category labels.

It can help discover themes, but a cluster is not automatically a business category. A clustering algorithm may group tickets by language, writing style, or ticket length when the business cares about support intent.

The learning objective must align with the real decision being made.

4. Representing text as features

Most classical models cannot directly operate on raw text. We must transform the text into numeric features.

A common representation is TF-IDF. It gives more importance to terms that are frequent in one document but not frequent across every document.

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(
    ngram_range=(1, 2),
    min_df=2,
)

features = vectorizer.fit_transform(ticket_texts)

Including bigrams allows phrases such as reset password and charged twice to become features.

This representation has limitations:

Similar meanings with different words remain separate.
Word order is represented only locally.
Sarcasm and implicit intent are difficult.
Vocabulary learned during training may not cover new terminology.

An embedding instead maps an input to a dense numeric vector intended to preserve useful semantic relationships.

"I cannot sign in"
"My login no longer works"
"The system rejects my credentials"

These sentences may receive nearby vectors even though they share few exact words.

Embeddings power:

Semantic retrieval
Clustering
Similarity search
Duplicate detection
Classification
RAG systems

An embedding is not a perfect representation of meaning. Its behaviour depends on its training data and objective. Similarity can also capture topic, style, language, or demographic correlations that are irrelevant to the current task.

5. Training, validation, and test data

Evaluating a model on examples it already learned from tells us whether it can remember its training data—not whether it can handle new data.

The dataset is therefore divided into separate subsets.

Split	Purpose
Training set	Learn model parameters
Validation set	Select features, model settings, and thresholds
Test set	Estimate final performance on unseen data

A typical split might be:

70% training
15% validation
15% test

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(
    texts,
    labels,
    test_size=0.30,
    stratify=labels,
    random_state=42,
)

X_validation, X_test, y_validation, y_test = train_test_split(
    X_temp,
    y_temp,
    test_size=0.50,
    stratify=y_temp,
    random_state=42,
)

The test set should not influence:

Feature selection
Hyperparameter selection
Prompt design
Threshold selection
Model choice
Error-driven changes

Repeatedly checking the test score while improving the model indirectly turns the test set into another validation set.

For production tickets, a random split may also be unrealistic. If the model will predict future tickets, a time-based split is often stronger:

January–April → training
May → validation
June → test

This exposes changes in products, terminology, customer behaviour, and issue frequency.

6. How a model learns

The model makes predictions using its current parameters. A loss function measures how wrong those predictions are.

Training repeatedly performs three conceptual steps:

Make predictions.
Measure loss.
Adjust parameters to reduce future loss.

For classification, the loss should penalize assigning low probability to the correct class. For regression, a loss might measure the distance between predicted and actual values.

Loss is a training objective, not automatically the final business metric.

A model can have a lower training loss while producing no useful business improvement. For example, it might improve common categories while becoming worse at detecting rare, urgent security tickets.

Optimization is the mechanism used to search for parameter values that reduce loss. Gradient-based optimization estimates how changing each parameter would affect the loss and moves the parameters in a better direction.

The important distinction is:

Parameters are learned from training data.
Hyperparameters are chosen by the developer or training process.

Examples of parameters:

Logistic-regression coefficients
Neural-network weights
Learned decision boundaries

Examples of hyperparameters:

Regularization strength
Tree depth
Learning rate
Number of training iterations
Embedding model
Classification threshold

Hyperparameters should be selected using validation data, not test data.

7. Generalization: the real objective

The goal of training is not to achieve the lowest possible loss on known examples. It is to perform well on new examples drawn from the environment where the model will be used.

This ability is called generalization.

Underfitting

A model underfits when it cannot capture enough of the underlying pattern.

Signs include:

Poor training performance
Poor validation performance
Similar errors across both sets

Possible causes:

Features contain too little information.
The model is too limited.
Training stopped too early.
The labels cannot be predicted from the available input.

Overfitting

A model overfits when it learns details specific to the training data that do not transfer to new data.

Signs include:

Excellent training performance
Considerably worse validation performance
Increasing validation loss while training loss continues to fall

Possible causes:

The model is too flexible for the available data.
The dataset is small.
Training runs too long.
Examples contain accidental shortcuts.
Hyperparameters were repeatedly tuned against one validation set.

Bias and variance

Bias is error caused by assumptions that are too restrictive. A high-bias model misses real patterns and tends to underfit.

Variance is sensitivity to the particular training sample. A high-variance model may learn unstable patterns and overfit.

The practical goal is not to eliminate both. It is to find a useful balance.

Regularization

Regularization discourages unnecessarily complex solutions.

It can include:

Penalizing large parameter values
Limiting tree depth
Dropping neural-network activations during training
Stopping training before validation performance declines
Adding more representative data

For logistic regression:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(
    C=1.0,
    class_weight="balanced",
    max_iter=1000,
)

A smaller C applies stronger regularization in scikit-learn’s logistic regression.

Cross-validation

When the dataset is small, a single validation split may give an unstable result. Cross-validation trains and evaluates the model across several different folds.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    model,
    training_features,
    training_labels,
    cv=5,
    scoring="f1_macro",
)

Cross-validation improves confidence in model selection, but it does not replace a final untouched test set.

8. Scaling, imbalance, and leakage

Feature scaling

Some algorithms are sensitive to the numerical scale of their inputs.

Suppose a model receives:

ticket_length: 1–5,000
customer_age_days: 1–2,000
sentiment_score: -1–1

Large numeric ranges can dominate distance calculations or optimization.

Scaling transforms features into comparable ranges:

from sklearn.preprocessing import StandardScaler

Scaling matters especially for:

K-nearest neighbours
Support-vector machines
Gradient-based linear models
Neural networks

Tree-based models are generally less sensitive to it. TF-IDF vectors are also commonly normalized already.

The scaler must be fitted only on training data. Fitting it on the entire dataset allows validation and test information to influence training.

Class imbalance

Suppose ticket categories have this distribution:

technical_issue: 70%
billing: 20%
account_access: 9%
security_incident: 1%

A model that never detects security incidents can still obtain 99% accuracy on that binary task.

Possible responses include:

Class-weighted loss
Over- or undersampling
Collecting more minority examples
Per-class thresholds
Separating critical cases into another detector
Using precision-recall metrics instead of accuracy

The right response depends on error cost. Automatically escalating every ticket may detect all security incidents, but it would overwhelm the security team.

Data leakage

Leakage occurs when training uses information that would not genuinely be available at prediction time.

Examples include:

Using the support agent’s final resolution notes to predict the initial category
Including a field written after escalation
Creating duplicates across training and test sets
Fitting the vectorizer on the complete dataset
Randomly splitting tickets from the same conversation across sets
Using future tickets to predict past outcomes

Leakage often produces impressive evaluation results and disappointing production performance.

A useful question is:

At the exact moment this prediction is made in production, would this value already exist?

If the answer is no, the feature is probably leaking future information.

9. Measuring classification performance

Accuracy

[ \text{Accuracy} = \frac{\text{Correct predictions}}{\text{All predictions}} ]

Accuracy is useful when classes and error costs are reasonably balanced. It becomes misleading when one category dominates.

Precision

[ \text{Precision} = \frac{\text{Correct positive predictions}}{\text{All positive predictions}} ]

Precision answers:

When the model predicts this category, how often is it right?

High precision matters when false alarms are expensive.

Recall

[ \text{Recall} = \frac{\text{Correct positive predictions}}{\text{All actual positives}} ]

Recall answers:

Of all examples belonging to this category, how many did the model find?

High recall matters when missing a case is expensive.

F1 score

F1 combines precision and recall into one number. It is useful for comparison but hides which kind of error changed.

For multiclass classification:

Macro F1 gives every class equal importance.
Weighted F1 weights classes by their frequency.
Micro F1 combines decisions across all examples.

Macro F1 is valuable when minority categories matter.

from sklearn.metrics import classification_report, confusion_matrix

predictions = model.predict(X_test)

print(classification_report(y_test, predictions))
print(confusion_matrix(y_test, predictions))

For ranking and retrieval systems, other metrics are needed:

Precision@k
Recall@k
Mean reciprocal rank
NDCG
Retrieval success rate

A metric is useful only when it represents the production decision and its failure costs.

10. Baselines before sophisticated models

A baseline establishes the minimum performance a proposed system must beat.

Useful baselines include:

Always predict the most common category.
Use the existing keyword rules.
Reuse the current manual routing process.
Retrieve the nearest labelled example.
Use a simple TF-IDF logistic-regression model.

Without a baseline, a complicated model can appear impressive while adding no measurable value.

The baseline comparison should include more than accuracy:

Per-class precision and recall
Latency
Cost
Operational complexity
Interpretability
Failure severity

A slightly more accurate system may not be worth deploying if it is much slower, more expensive, and harder to debug.

11. Error analysis

A metric tells us how often the model fails. Error analysis tells us why.

After evaluation, collect incorrect predictions and group them by meaningful causes:

Error category	Example
Ambiguous ticket	“My account was charged but I cannot log in”
Missing context	Ticket refers to an attachment
Label inconsistency	Similar tickets assigned different categories
Rare wording	New internal product name
Multiple intents	Cancellation combined with refund request
Language issue	Mixed Hindi and English
Product change	New feature absent from training data
Shortcut learning	Model relies on customer tier

Error analysis may reveal that the correct solution is not a larger model. It might be:

Better label definitions
Multilabel classification
Additional context
A human-review state
New training examples
A product-specific routing stage

Models produce scores; systems must decide what to do with uncertainty. A low-confidence prediction can be sent for manual review instead of being forced into a category.

12. Distribution shift

Training assumes that future data will resemble the data used for training. Distribution shift occurs when this assumption stops holding.

Examples include:

A new product generates unfamiliar ticket types.
Customers begin using AI-generated support messages.
The company expands into another country.
Category definitions change.
A mobile release causes a sudden concentration of one issue.
Agents change how they label tickets.

Shift can affect:

Input language
Feature frequencies
Class proportions
The relationship between inputs and labels

A time-based test can simulate some shift:

train = tickets[tickets["created_at"] < "2026-05-01"]
test = tickets[tickets["created_at"] >= "2026-05-01"]

Production monitoring can look for:

Changes in category frequency
Changes in vocabulary or embedding distributions
Increasing low-confidence predictions
Declining accuracy on reviewed samples
Rising human override rates
Performance differences by product or language

Detecting input change is not the same as proving performance degradation. Labels or human review are still needed to measure correctness.

13. Transfer learning, fine-tuning, and inference

Transfer learning

Transfer learning reuses representations learned on a broad dataset for a new task.

Instead of learning language from support tickets alone, we can use a pretrained embedding model and train a small classifier on top of its vectors.

This usually requires less task-specific data than training a language representation from scratch.

Fine-tuning

Fine-tuning updates some or all parameters of a pretrained model using task-specific examples.

Fine-tuning may be justified when:

The task is stable and clearly specified.
Enough high-quality examples exist.
Prompting or simpler models do not meet requirements.
The same behaviour is required repeatedly.
Expected quality gains justify training and maintenance costs.

It may not be justified when:

Labels are inconsistent.
Requirements change frequently.
Few representative examples exist.
Retrieval or prompt improvements solve the problem.
A classical classifier already meets the target.
The real failure comes from missing context.

Fine-tuning does not automatically teach a model changing factual knowledge. Frequently changing knowledge is often better supplied through retrieval.

Inference

Training learns the model’s parameters. Inference applies the frozen model to new inputs.

A production inference pipeline must reproduce the same preprocessing used during training:

flowchart LR
    A["Ticket"] --> B["Text preprocessing"]
    B --> C["Feature transformation"]
    C --> D["Model"]
    D --> E["Probabilities"]
    E --> F["Threshold or review policy"]

Versioning only the model file is insufficient. The deployed unit should include:

Text normalization
Vectorizer or tokenizer
Model parameters
Label mapping
Thresholds
Schema
Training-data version

14. Offline and online evaluation

Offline evaluation uses a fixed dataset before deployment.

It is useful for:

Reproducible comparison
Regression testing
Per-class analysis
Adversarial and shift testing
Fast iteration

However, offline data may not represent how the system changes user or employee behaviour.

Online evaluation observes the deployed system.

Possible measurements include:

Agent correction rate
Ticket reassignment rate
Resolution time
Escalation rate
Customer satisfaction
Operational cost
A/B test outcomes

Online improvements do not automatically prove model improvements. UI changes, staffing, seasonality, or product incidents may influence the same outcome.

A safe deployment can begin with shadow mode: the model makes predictions, but humans continue making the actual routing decisions. The two can then be compared.

15. Feedback loops

A deployed model can change the data that will later be used to retrain it.

Suppose the model routes a ticket to billing. The billing agent accepts the existing category without checking it. That category later becomes a training label.

The system is now learning from its own previous prediction.

Other feedback loops include:

Low-ranked documents receive fewer clicks, creating less positive feedback.
Automatically rejected requests never receive detailed human labels.
Recommended actions become more common because they were recommended.
Support agents adapt their wording to match the classifier.

Feedback data should record its provenance:

Was the label created independently?
Did a human review the prediction?
Was the model prediction visible to the reviewer?
Was the label changed?
Which model version produced the original decision?

Without provenance, retraining can amplify earlier mistakes.

Practical Project: Support-Ticket Classifier

1. Inspect the dataset

Begin by checking structure, missing values, duplicates, category counts, and example quality.

import pandas as pd

tickets = pd.read_csv("support_tickets.csv")

print(tickets.head())
print(tickets.info())
print(tickets["category"].value_counts())
print(tickets.isna().sum())
print(tickets.duplicated(subset=["text"]).sum())

Manually inspect examples from every category:

for category, group in tickets.groupby("category"):
    print(f"\nCATEGORY: {category}")
    print(group["text"].sample(min(5, len(group)), random_state=42).tolist())

Before modelling, define:

What each category means
Whether multiple categories are allowed
What context is available at inference time
How ambiguous tickets should be handled
Which mistakes are most expensive

2. Implement baselines

Create at least two baselines:

Always predict the majority category.
Apply existing keyword rules.

from sklearn.dummy import DummyClassifier

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)

The learned model must beat these baselines on the metrics that matter.

3. Build the first model

A TF-IDF and logistic-regression pipeline is a strong, understandable starting point.

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    (
        "tfidf",
        TfidfVectorizer(
            ngram_range=(1, 2),
            min_df=2,
            max_df=0.95,
        ),
    ),
    (
        "classifier",
        LogisticRegression(
            max_iter=1000,
            class_weight="balanced",
        ),
    ),
])

pipeline.fit(X_train, y_train)

Because preprocessing and prediction are in one pipeline, the vectorizer is fitted only on training data and the same transformation is reused during inference.

4. Evaluate it

from sklearn.metrics import classification_report

validation_predictions = pipeline.predict(X_validation)

print(
    classification_report(
        y_validation,
        validation_predictions,
        digits=3,
    )
)

Inspect:

Macro F1
Per-class precision
Per-class recall
Confusion matrix
Performance of critical categories
Difference between training and validation performance

Do not change the model based on test-set results.

5. Handle imbalance deliberately

Compare:

No class weighting
class_weight="balanced"
Additional minority-class data
Threshold changes
Manual review for uncertain cases

Do not apply resampling before splitting the dataset. Otherwise, copies or synthetic relatives of the same example can cross split boundaries.

6. Perform error analysis

Create an error table:

analysis = pd.DataFrame({
    "text": X_validation,
    "actual": y_validation,
    "predicted": validation_predictions,
})

errors = analysis[analysis["actual"] != analysis["predicted"]]
errors.to_csv("validation_errors.csv", index=False)

Annotate errors by cause. Improve one well-supported failure category at a time and rerun the evaluation.

7. Test distribution shift

Evaluate on a newer time period, another product, or another customer segment.

Compare performance across:

Old versus recent tickets
English versus mixed-language tickets
Product lines
Customer plans
Short versus long tickets
Common versus newly introduced categories

A useful aggregate score can hide a severe failure in one segment.

8. Version the complete model

Record:

{
  "model_version": "ticket-classifier-1.0.0",
  "training_data_version": "tickets-2026-07-15",
  "code_commit": "abc123",
  "features": "tfidf-unigram-bigram",
  "algorithm": "logistic-regression",
  "class_weight": "balanced",
  "decision_policy": "argmax",
  "label_schema_version": "2",
  "validation_macro_f1": 0.84
}

Persist the entire pipeline:

import joblib

joblib.dump(pipeline, "ticket_classifier_v1.joblib")

9. Expose an inference API

from fastapi import FastAPI
from pydantic import BaseModel, Field
import joblib

app = FastAPI()
model = joblib.load("ticket_classifier_v1.joblib")


class TicketRequest(BaseModel):
    text: str = Field(min_length=1, max_length=10_000)


@app.post("/classify")
def classify_ticket(ticket: TicketRequest):
    probabilities = model.predict_proba([ticket.text])[0]
    classes = model.classes_

    best_index = probabilities.argmax()
    label = classes[best_index]
    confidence = float(probabilities[best_index])

    return {
        "label": label,
        "confidence": confidence,
        "model_version": "ticket-classifier-1.0.0",
        "requires_review": confidence < 0.60,
    }

The probability should be treated carefully. A value of 0.90 is not necessarily a calibrated 90% chance of correctness. Calibration must be evaluated separately.

10. Create a monitoring plan

Monitor four different layers:

Layer	Measurements
Service	Latency, errors, throughput, availability
Inputs	Missing text, length, language, vocabulary or embedding shift
Predictions	Category distribution, confidence, review rate
Outcomes	Corrections, reassignments, precision, recall, resolution time

Store enough information to connect:

request → model version → prediction → human correction → final outcome

Use sampled human review to obtain ongoing ground-truth labels. Input drift alone cannot tell you whether the model is still correct.

Traditional ML versus an LLM Classifier

Dimension	Traditional classifier	LLM classifier
Initial labelled data	Usually required	Can work zero-shot or few-shot
New category adoption	Retraining may be needed	Prompt changes may be sufficient
Latency	Usually low	Usually higher
Per-request cost	Very low	Higher
Output consistency	High	More variable
Interpretability	Often easier	Reasoning may sound plausible but be unreliable
Complex language	Limited by representation	Usually stronger
Deployment	Can run locally	Often depends on model infrastructure
Privacy	Easier to keep internal	Depends on deployment and provider
Context use	Requires engineered features	Can directly use instructions and context
Version stability	Strongly controlled	Provider models may change
Fine-tuning	Often inexpensive	Potentially expensive and operationally complex

A useful comparison should evaluate both systems on the same frozen dataset and label definitions.

The LLM classifier should use:

A fixed prompt version
Structured output
Deterministic label validation
Low temperature where supported
Recorded model and provider versions
The same per-class metrics
Latency and cost measurements

The best architecture may be hybrid:

Use a classical model for common, high-confidence cases.
Send uncertain or novel cases to an LLM.
Escalate high-risk ambiguity to a human.
Record corrections for future evaluation.

The decision should come from measured constraints—not from assuming that newer or larger models are automatically better.

The ML Project Lifecycle

A reliable machine-learning project can be reconstructed as:

Define the production decision.
Define labels and failure costs.
Inspect how data was generated.
Establish simple baselines.
Create leakage-safe dataset splits.
Build the simplest reasonable representation and model.
Train on training data.
Select settings using validation data.
Analyze errors, not only aggregate metrics.
Evaluate once on an untouched test set.
Test realistic distribution shifts.
Version data, preprocessing, model, threshold, and schema.
Deploy with validation and fallback behaviour.
Monitor inputs, predictions, outcomes, and segments.
Collect independently reviewed feedback.
Retrain only when evidence justifies it.

The central lesson is simple:

A machine-learning system is not merely a model. It is a decision process whose behaviour is shaped by data, objectives, evaluation, deployment policy, and feedback.

This same principle applies to embeddings, retrieval systems, fine-tuned models, LLM judges, and autonomous agents. The model may be probabilistic, but the engineering around it must make its assumptions, limits, and failures observable.