Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Twu31 
posted an update 11 days ago
Post
85
Can a Jev-style decision interface work on EEG when the questions are asked in words? In our tests, only for questions the model was trained on.

A small head over frozen EEG features (encode a window once, answer several typed questions about it) was asked each question by a question number, a label template or a description. On SSVEP (BETA, 70 people), asking a seen question with a label template instead of its number cost −5.96 pp of balanced accuracy on a plain spectrum (interval −6.84 to −5.08 pp), the 2 pp margin not met. On flicker frequencies the head was never trained on, it reached 28.7%, where training-free CCA reached 80.9% (chance 12.5%).

Results and limits: https://bci.report/topics/questions-in-language/
What "Jev-style" means here: https://bci.report/jev-style/
Query the numbers over MCP: Twu31/bci-report-explorer
Related write-up: https://huggingface.co/blog/Twu31/one-eeg-encoder-several-questions
dipankarsarkar

The ordering on unseen frequencies is the headline for me: words 28.7%, a numeric code 41.5%, the one-hot head's neighbour average 44.3%, CCA 80.9%.

So text is not the bottleneck it looks like.
Every query goes through a head that gets 32.1% on the seen 32-way, where CCA gets 65.5%. Asking in words can only lose from there.

CCA asks with a reference: sinusoids at the queried frequency and its harmonics. The question carries the physics.

Did you try that as the FiLM vector? The plain spectrum of the reference signal, in place of text or a number.
That would split "the head can't condition" from "the query can't encode a frequency". Which way would you bet?

·
Twu31

Thanks for the comment! Yes, those are the plain-spectrum numbers. On frozen CBraMod features the same three arms all sit around 18–21%, so I think your ceiling point holds there too.

We didn't try the reference signal as the query. The numeric code is the closest thing we have, and it only encodes the fundamental, no harmonics.

If I had to bet, I'd say it's mostly the head: even with one-hot questions it only gets 32.1% on the seen 32-way, while CCA gets 65.5%. Honestly not sure though, and your idea is a really clean way to tell the two apart, so we'll add the reference-spectrum arm and test it. I'll post back here when we have results.
dipankarsarkar

Thanks, and good to know the frozen-CBraMod arms all sit at 18 to 21.

The seen set already settles most of it. One-hot is a perfect code, and the head still gets 32.1 where CCA gets 65.5. On seen frequencies the head is the whole gap. No query fixes that.

Unseen is the open half. One-hot has no way to name a new frequency, so 44.3 is a neighbour read-off, not a ceiling on the query. The reference spectrum names it, harmonics included.

My bet, on record before your run: the reference-spectrum arm beats 44.3 on unseen. 70 percent. Above 60: 30 percent.

If it also lands near 32 on seen, the head caps both, and I lose on the mechanism even if I win on the number.

Which number would change your mind?

·
Twu31

Following up on this: we ran the reference-spectrum arm (fundamental plus 2nd and 3rd harmonics, on the same 7–48 Hz bins as the EEG spectrum, everything else unchanged).

On your bet: unseen came in at 39.0%, below the 44.3% neighbour read-off and also below the numeric code's 41.5% (paired diff −2.5 pp, interval −3.5 to −1.5). CCA is still at 80.9%. On seen it landed at 31.5%, basically on top of one-hot's 32.1% (within our ±2 pp equivalence margin). CBraMod features show the same pattern.

So by your own framing the head caps both, and I'd mostly agree. I wouldn't put all of it on the head, though: a fixed template match on the same spectra, with no training, gets only 28.8% on the unseen 8-way, and CCA works on the raw multichannel signal, with timing and phase info the magnitude spectra throw away.

What would change my mind: if a stronger head, say a matching head, on the same spectra still sat near 32% on seen, I'd blame the features rather than the head. That's the obvious next thing to try. Thanks for putting a number on it.
dipankarsarkar

I lost that one, cleanly. 39.0 on unseen, under the 44.3 I said it would beat, and 31.5 on seen. Thanks for running it and for holding me to the number.

Your template match is the line I'd build on. 28.8 on the unseen 8-way with no training, where chance is 12.5. So the magnitude spectra do carry the frequency, and a trained head on them reaches 39 to 44. CCA gets 80.9 with phase and channels the spectra drop. That gap is not the head's to close.

The cheapest matching head is no query at all: a plain 32-way softmax on the same spectra, seen classes only. It cannot do unseen, but it prices the seen half. Near 32, the features cap it. Near 65, the conditioning does.

Do you already have that number from a baseline run?

·
Twu31

Thanks for being so straightforward about it. Putting the number down first made the result more useful. And yes, we ran the softmax check.

On the same magnitude spectra, a plain linear 32-way softmax gets 35.9% on seen (interval 32.3 to 39.3). The original hidden-layer head trained the same way gets 31.5%. So near 32, not near 65. By your framing the features cap it, and the matching head says the same: 34.6% with one-hot questions, a small gain at best.

We also swapped the magnitude spectra for the complex spectrum (real and imaginary parts, so phase and cross-channel timing are kept). Seen jumps to 50% with the original head and 56% with the matching head. That isn't a pure phase effect, since scaling and dimension change too, but it points the same way.

Unseen is still the hard part. On complex features, a phase-aware reference query improves on the magnitude query by about 16 pp, but it only reaches 22–25% on the unseen 8-way, below the 39% from the magnitude-spectrum reference and nowhere near CCA's 80.9%. Dropping the harmonics also hurts (−13 pp on unseen).

So seen looks feature-limited, and transfer to new frequencies is the open problem. We're still a long way from CCA's 80.9% on unseen, so if you have other ideas worth trying, I'd love to hear them. Thanks again; this thread shaped the whole follow-up.
dipankarsarkar

Phase helps seen and hurts unseen. That split is the clue.

On complex spectra, seen goes to 50 to 56. Unseen falls to 22 to 25, against 39 on magnitude.
A head trained on complex spectra can store each seen class's absolute phase. That is a template, and a template says nothing about a frequency it never saw.

CCA never needs the absolute phase. Its reference is a sin and cos pair per harmonic, so the fit is the same at any stimulus phase. What it uses is relative timing across channels.

Cheap test: same complex spectra, a random onset shift per training trial, up to one stimulus period. That scrambles absolute phase and keeps cross-channel phase.
My bet: unseen clears 39. 55 percent.
If phase is coded per target, seen should drop some, since phase is then part of the label.

Is it? 40 targets with harmonics up to 48 Hz reads like the JFPM benchmark.

·
Twu31

Yes, BETA uses joint frequency-phase modulation: 40 targets from 8 to 15.8 Hz in 0.2 Hz steps, with a 0.5π phase step between neighbouring frequencies. The 48 Hz is just our feature band, not the stimulus design. So phase is part of the label for the seen targets, which fits your story: on complex spectra the head can learn each seen target's phase as a template, and that doesn't carry over to new frequencies.

Nice test, and a cheap one. We can do the shift on the cached complex spectra directly: a random per-trial delay multiplies every bin by the same linear phase ramp on all channels, so absolute phase gets scrambled and cross-channel phase is kept. We'll run it with the magnitude reference and the numeric code as queries, and I'll report back on both halves: whether unseen clears 39, and whether seen drops.
dipankarsarkar

I lost that one too. 30.6 on unseen, under the 39 I bet it would clear. Seen fell, as called, but that was the cheap half.

The number that explains it for me is one-hot at 3.7 under the shift. Chance on the 32-way is 3.1.
The shift does not touch magnitude. Your heads got about 32 to 36 on magnitude spectra, so a head that took |z| would have kept about that. It kept nothing.

Under a random delay, any linear read of Re and Im has mean zero for every class. The class lives in second-order terms: |z|², and z_i · conj(z_j) across channels, where the shared delay cancels. The head has to learn a modulus from data, and it didn't.

Both of your training-free numbers are second-order. Subspace matcher 73.2 on unseen, CCA 80.9. That 7.7 gap is about the 8 pp you measured for the Hann taper alone.

So I'd hand the head the second-order terms instead of asking it to learn them. For each candidate frequency, the channel-by-channel matrix at its fundamental and harmonics. Score every candidate with one shared set of weights, nothing per frequency. CCA has no per-frequency parameters, which is why an unseen frequency is not new to it.

My bet: that head clears 39 on unseen. 60 percent. With the shift on, it should also pull seen one-hot back into the 32 to 36 band.

Is your subspace matcher built on that matrix, or on a projection onto the reference?

·
Twu31

It's a projection onto the reference: basically CCA (top canonical correlation between the 8-channel span and the sine–cosine reference span), computed on the same Hann-windowed, cropped complex coordinates, with channel weights fit per trial. It lands 0.2 pp from Hann-windowed CCA in the time domain, so yes, the gap to 80.9 is basically the taper.

The two views are close, though. With near-orthogonal sine–cosine references, CCA's squared score is roughly the top eigenvalue of Σ_h Re(z_h z_hᴴ) at the candidate's harmonics, whitened by the channel covariance. So it is your matrix, just with a fixed rule on top instead of learned weights.

The closest learned thing we've run is a ridge scorer with one shared set of weights for every candidate, but on the diagonal only: each channel's energy fraction at each harmonic, 24 features. It got 57.4% on unseen (53.2 to 62.1) and 34.9% on seen. So the diagonal alone clears 39 and seen lands in your 32 to 36 band, but it's still 23 pp under CCA. What it's missing is the cross-channel terms, which is exactly your proposal. We'll run the full-matrix version and report back.
dipankarsarkar

Then my last bet was too easy. The diagonal alone gets 57.4 on unseen, so the full matrix will clear 39 almost by construction. I'll count the idea, not the bet.

Your eigenvalue line is the useful part. It says where the other 23 pp live.

A shared linear scorer on the matrix is tr(W M) for one fixed W. That is a fixed spatial filter, the same on every trial.
CCA takes the top eigenvalue after whitening by that trial's covariance. Channel weights fit per trial, as you said.
So the gap splits in two: the cross-channel terms, and the per-trial max over filters. The raw-entry ridge gets the first and none of the second.

Two things I'd try alongside it:

  1. Normalize the off-diagonals the way your 24 features already are: coherence, |M_ij| over sqrt(M_ii M_jj). Raw entries carry per-subject scale.
  2. Give the head the top few eigenvalues of the whitened matrix per harmonic, with shared weights. Those are CCA's own statistic, rotation-invariant, and leave only the mix across harmonics to learn.

My bet: raw entries land under 70 on unseen. 65 percent.
The eigenvalue head lands within 5 of 80.9. 50 percent.

Are the test subjects held out from training? If yes, a fixed W has to fit every head shape at once, and that is where I'd expect it to stall.
In this post