iwantcoding.com
🔥 Daily 👥 Rooms 🏆 Top Log in Sign up

Losses

By · Reviewed & last updated · Part of PyTorch Tutorial

PyTorch ships every loss as an nn.Module — CrossEntropyLoss, MSELoss, BCEWithLogitsLoss, and friends. Custom losses are ordinary subclasses or just functions.

The losses you reach for + a custom one

EXAMPLE
import torch
import torch.nn as nn
import torch.nn.functional as F

# 1) Multiclass classification — logits + integer labels (most common)
ce = nn.CrossEntropyLoss()
logits = torch.randn(8, 10)                    # batch=8, classes=10
targets = torch.randint(0, 10, (8,))
loss = ce(logits, targets)

# Class imbalance — pass weights
weights = torch.tensor([1.0] * 9 + [5.0])      # last class is rare
ce = nn.CrossEntropyLoss(weight=weights)

# Label smoothing — better generalisation
ce = nn.CrossEntropyLoss(label_smoothing=0.1)

# 2) Binary classification — logits + 0/1 labels
bce = nn.BCEWithLogitsLoss()                   # stable; never feed probabilities
logits = torch.randn(8, 1)
targets = torch.randint(0, 2, (8, 1)).float()
loss = bce(logits, targets)

# 3) Regression
mse = nn.MSELoss()                              # squared error
mae = nn.L1Loss()                               # absolute error
huber = nn.SmoothL1Loss(beta=1.0)               # combination — robust to outliers

# 4) Embedding / contrastive
cos   = nn.CosineEmbeddingLoss()
triplet = nn.TripletMarginLoss(margin=1.0)

# 5) Sequence — variable length output
ctc = nn.CTCLoss()

# 6) Custom — focal loss
class FocalLoss(nn.Module):
    def __init__(self, alpha=0.25, gamma=2.0):
        super().__init__()
        self.alpha = alpha
        self.gamma = gamma

    def forward(self, logits, targets):
        bce = F.binary_cross_entropy_with_logits(logits, targets, reduction='none')
        p = torch.sigmoid(logits)
        pt = torch.where(targets == 1, p, 1 - p)
        w = self.alpha * (1 - pt) ** self.gamma
        return (w * bce).mean()

# 7) Multi-task — weight + sum component losses
class_loss = ce(class_logits, class_targets)
reg_loss   = mse(score_pred, score_targets)
loss = class_loss + 0.3 * reg_loss
loss.backward()

Why it matters

BCEWithLogitsLoss > sigmoid + BCELoss. Same math, but numerically stable for large positive or negative logits. The same lesson applies to CrossEntropyLoss + raw logits.

Tip: Tweak the snippet with Try it Yourself », then sit the quiz at the bottom of the page.

Example

Example
import torch.nn as nn
ce  = nn.CrossEntropyLoss()
bce = nn.BCEWithLogitsLoss()
mse = nn.MSELoss()
Try it Yourself »

Discussion

Loading…