Implementation Guide

1. Overview & pipeline

OnPoint is a two-stage framework:

1. Train a point-supervised offline teacher on full videos (HR-Pro for THUMOS'14, TSASPC for EGTEA/HOI4D-O).
2. Freeze the teacher and train an online student (built on OAT) with multi-level distillation plus direct point supervision.

Video features
    │
    ├─► [Offline teacher] ──► CAS sequence A
    │                         Pseudo segments (s̃, ẽ, ã, p̃)
    │
    └─► [Online student, sliding window at each t]
            Transformer encoder → F_t
                ├─ CASS predictor → Â_t          (distill from A_t^teacher)
                ├─ Anticipation head → ŷ         (distill from future CAS)
                ├─ Anchor decoder → Z            (actionness-calibrated)
                │     ├─ Instance cls/reg heads  (supervised by pseudo segments)
                │     └─ Point cls/reg heads     (supervised by point GT)
                └─ ONMS post-processing → final proposals

Recommended starting point: fork OAT for the online backbone and anchor pipeline, then add the four OnPoint modules described below.

2. Problem definition (POTAL)

Point-Supervised Online Temporal Action Localization (POTAL) detects action segments in a streaming video using only one temporal point + class label per action instance at training time.

• Input at time t: partial video V_1:t (no future frames).
• Output: proposals (ŝ, ê, â, p̂) with start/end times, class, confidence.
• Proposals are emitted online and cannot be revised once published.
• Goal: recover the full action set Ψ = {(ŝ_k, ê_k, â_k)} sequentially.

3. Phase 1 — Data & features

3.1 Datasets

ActivityNet v1.3 is excluded from OnTAL evaluation (videos usually contain one long action).

3.2 Pre-extracted features

Project all features to dimension D = 1024 before the online transformer encoder (match OAT feature dim).

4. Phase 2 — Offline teacher

5. Phase 3 — Generate pseudo labels for online training

THUMOS'14 (HR-Pro)

1. Run HR-Pro on each training video to obtain CAS A.
2. Extract intermediate reliable proposals (not final merged output) as pseudo segment GT for instance distillation.
3. Slice CAS to each online window: A_t^teacher = A[t−W+1 : t].
4. Slice future CAS for anticipation: A[t+1 : t+W′] → binarize with threshold 0.5 → multi-hot target y.

EGTEA / HOI4D-O (TSASPC)

1. Run TSASPC to obtain per-frame CAS.
2. Build pseudo segments by grouping temporally adjacent frames with the same predicted class.
3. Align pseudo segments and CAS slices to each sliding window as above.

Anchor–pseudo matching (instance supervision)

Follow OAT's anchor assignment: match each pseudo segment to anchors in the current window by temporal IoU / center overlap. Positive anchors receive classification + regression targets from the matched pseudo segment (same protocol as fully supervised OAT, but labels come from teacher).

6. Phase 4 — Online student architecture

At each timestep t, form window X_t ∈ ℝ^W×D from the last W feature vectors. Pass through:

6.1 Transformer encoder (from OAT)

• 3 transformer blocks, 8 attention heads each, hidden dim D=1024.
• Output window features F_t ∈ ℝ^W×D.

6.2 CASS predictor (new)

• Two-layer MLP per frame: Linear(D→D) → ReLU → Linear(D→C+1).
• Output Â_t ∈ ℝ^W×(C+1).
• Use L2 (MSE) loss against teacher CAS slice (do not normalize to a probability simplex).

6.3 Anticipation head (new)

• Linear(D → D/4) on F_t, flatten, then FC → ReLU → FC → Sigmoid.
• Output ŷ ∈ [0,1]^C+1 (multi-label).
• Target: binarize teacher CAS over next W′ frames (threshold 0.5) → y ∈ {0,1}^C+1.
• Loss: binary cross-entropy over action classes.

Use fixed W′ per dataset (see hyperparameter table).

6.4 Actionness-calibrated anchor decoder (new)

Start from OAT's anchor decoder (5 transformer blocks, 4 heads). Replace standard cross-attention with calibrated attention:

1. Initialize learnable anchor queries Q ∈ ℝ^M×D (M anchors with fixed temporal scales).
2. Actionness from CASS background channel: r = 1 − σ(A_t[:, C+1]).
3. Calibration: r̄ = r + log(r) (element-wise; clamp r for numerical stability).
4. K = Linear(F_t), V = Linear(F_t).
5. Each decoder layer:
   • Self-attention on Q.
   • Cross-attention: logits = QK^⊤/√D + r̄^⊤, α = Softmax(logits), context = α·V.
   • Q ← FeedForward(Q + context).
6. Output anchor features Z = Q.

6.5 Instance prediction heads (from OAT)

On each anchor feature in Z:

• Classification head: anchor class + foreground/background (follow OAT).
• Regression head: temporal offsets/scales to refine anchor boundaries.
• Supervised by matched pseudo segments → L_ins = L_ic + L_ir (same as OAT).

6.6 Anchor-level point prediction (new)

For each anchor, two two-layer MLP heads (ReLU between layers), input = anchor embedding ∈ ℝ^D:

• Classification: cross-entropy — does this anchor contain a point label, and which class?
• Regression: predict normalized distance d̂ = |c_a − p| / l_a (anchor center, point time, anchor length).

Assign point labels to anchors whose temporal span contains the annotated point.

7. Phase 5 — Losses & training

7.1 Total objective

Recommended weights: α = β = δ = 1. Tune only γ (anticipation).

7.2 Training loop (per iteration)

1. Sample a training video and timestep t (or a batch of windows).
2. Load feature window X_t, teacher CAS slice, future CAS, pseudo segments, and point labels for that window.
3. Forward online student → compute all four losses.
4. Backpropagate L_total; teacher remains frozen.

7.3 Optimizer

• Adam, lr = 1e-4, weight decay = 1e-4, batch size = 128.
• Train until convergence using the same epoch count / schedule as OAT for the target dataset.

8. Phase 6 — Inference & evaluation

8.1 Online inference

1. Stream features frame-by-frame; maintain a buffer of the last W features.
2. At each t, run encoder + anchor decoder + instance heads only (CASS/anticipation/point heads used at train time; instance heads drive inference).
3. Decode anchors to proposals {(ŝ_k, ê_k, â_k, p̂_k)} following OAT.
4. Apply Online NMS (ONMS) from MATR codebase:
   • Suppress proposals with high temporal overlap with already-selected ones.
   • Remove proposals whose end time ê > current time t (wait for boundary confirmation).

8.2 Metrics

• Primary: mAP at tIoU thresholds {0.1, 0.2, 0.3, 0.4, 0.5} (and 0.6, 0.7 on THUMOS).
• A prediction is correct iff class matches and tIoU ≥ threshold.
• Secondary: instance-level F1 at tIoU = 0.5.

9. Distillation-free baseline

To ablate distillation, remove L_ins and train with anchor-level point supervision only (L_pnt as primary signal):

• At inference, use point prediction head: if any class activation > 0.1, treat anchor as proposal.
• Use anchor boundaries + regressed center offset as segment; apply ONMS.
• Without pseudo segments, the model cannot learn proper temporal scale, so performance is substantially lower than the full distillation-based model.

10. Hyperparameters

Per-dataset online settings

Shared online architecture

• D = 1024, encoder: 3 blocks × 8 heads, decoder: 5 blocks × 4 heads
• Batch size = 128, Adam lr = 1e-4, weight decay = 1e-4
• α = β = δ = 1; tune γ only

11. Environment & seeds

Hardware

• Training: NVIDIA L4, 24 GB, Driver 535.183.06
• Efficiency benchmark: RTX 4090 (same checkpoint)

Software

• Python 3.10.18, PyTorch 2.5.1, CUDA 11.8, TorchVision 0.20.1
• NumPy 2.0.1, SciPy 1.15.3, Pandas 2.3.1, Matplotlib 3.10.3, TensorBoard 2.19.0

Random seeds (single run per experiment)

• Online model: 52
• Offline HR-Pro: 0
• Offline TSASPC: 1538574472

Public code dependencies

12. Implementation checklist

Use this list to track a from-scratch re-implementation:

• ☐ Download dataset features and point labels for target benchmark
• ☐ Train / obtain checkpoint for offline teacher (HR-Pro or TSASPC)
• ☐ Pre-compute and cache per-video CAS and pseudo segments
• ☐ Implement sliding-window dataloader with teacher label alignment
• ☐ Port OAT transformer encoder + anchor instance heads
• ☐ Add CASS predictor + L2 distillation loss
• ☐ Add anticipation head + BCE on binned future CAS
• ☐ Modify anchor decoder cross-attention with r̄ = r + log(r) bias
• ☐ Add anchor-level point cls/reg heads + L1 regression
• ☐ Combine losses with α=β=δ=1, tune γ
• ☐ Integrate ONMS at inference
• ☐ Evaluate mAP@tIoU (and optionally instance-level F1)