Generative Adversarial Network for Neural Decoding

in #deep-learning7 years ago (edited)

Today I've been researching GAN's and how we might be able to use one for our problem. Right now I'm still focusing on fMRI data, but I hope to gain insight about developing models for calcium data as well. (Calcium imaging is a new neural imaging technique with far higher resolution.)

In Machine Learning most problems generalize to inferring latent variables from observable data (the feature space). Our case, inferring visual stimuli from fMRI data, is no exception. A discriminative model can accomplish this, but it can't do the reverse: that is, generate the data. A generative model, on the other hand, can do both.

A generative adversarial network simultaneously trains both a generative model G and a discriminative model D that compete with each other. While G trains on the training data, D's job is deciding whether data is from G or from actual training data. D and G participate in a so called zero-sum game, which means that the loss of one is the gain of the other and vice-versa.

Think of G as a animal of prey that evolves to be better and better at camouflage and D as a predator that becomes better at differentiating between prey and other surroundings. For our purposes, G trains on the neural imaging data, while D differentiates between actual neural imaging and imaging generated by G. 

To quote the original paper:

 The training procedure for G is to maximize the probability of D making a mistake. 

This is my idea for setting up a GAN for fMRI data:

Disregarding PCA for the moment, the generative model looks at the visual stimuli (the training labels if you will) and generates "fake" fMRI data which the discriminative model compares with real fMRI data. The problem is, that fMRI data has far too many dimensions so we use PCA for dimensionality reduction. 

PCA is basically singular value decomposition where the smaller singular values are thrown away. If you don't know what that means, think of it as a handy way to compress the fMRI data so we don't spend ages computing. Using PCA for dimensionality reduction on our GAN means that the generative model no longer generates actual fMRI data but the principal components of fMRI data. The discriminative model compares generated principal components with real principal components calculated from observed data.

I don't know if such a system will work or not. The challenge now is to implement such a network in Tensorflow and find out. If anyone has tips, suggestions or feedback please let me know. I'll keep you updated.

Sort:  

All right, there were somethings that I misunderstood about GANs when I wrote this post. First of all, GANs are for unsupervised learning (that is learning on unlabeled data). That means GANs are applicable when we aren't really sure what our latent space is. So, typically, the input for the generative network would be a noise vector. Second of all, my discriminative network would have no way of knowing what stimuli the fMRI data (whether real or counterfeit) corresponds to. So it could only find out if the generative network delivers plausible fMRI data in general and not for a specific stimuli. Another problem is that even if we have a trained generative network for generating fMRI data, it is unclear how this helps us with decoding. As far as I know, there's no way to just "reverse" a network like that. I need to look into this more, but for right now, I don't see how GANs are useful for our problem. Please let me know your thoughts. I'm debating with myself whether to take this post down...

This looks like an AI vs. AI fight :D

By the way, I think it would be very useful, for the readers, to add extra references where more information could be found. In addition, I am fully convinced this will bring you more rewards ;)

Thanks in advance for your consideration.

Thank you so much for the feedback! For this post I just used Goodfellow's original paper and wikipedia. Ill try to give more sources in future.

Thanks for letting me know. :)

Why not adding this information at the end or beginning of the post? You could for instance provide links to the sources to your readers.

Congratulations @trilo! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

You published your First Post
You got a First Vote
You made your First Comment

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!

Yay me! Do I only have a limited amount of upvotes? Does the power of an upvote dilute if I use more? I don't understand this website yet...

Every time you upvote something, your 24 hour supply of voting power is reduced. I think it drops by 2%. Every 24 hours it regenerates, I think by 20%. So if you want to start off each day at 100% voting power, limit yourself to 10 upvotes per day. On the other hand, new users start at such low reputation and Steem power anyway that you're not really losing anything by upvoting more than that. I mean, each ipvote only counts for a fraction of a penny, right? If you were a "whale," things would be different. Incidentally, you can use this URL to keep track of your voting power: https://steemd.com/@trilo.

Thanks man. Great community on here. :)

Hi @trilo, interesting article. I'm going to have to follow you now! Your interest in neuroscience fits with my interest in machine learning. Maybe we can compare notes now and then. I believe that our next great leaps in machine learning will be inspired by things we learn from biology.

Your proposal sounds interesting, but I suspect that the devil will be in the details. One concern is whether fMRI will give you enough granularity. I wonder how many neurons and synapses are represented by one pixel in the image.

You might want to try the experiment without principal component analysis in the beginning. That's the advice I've heard from data scientists in the field. PCA is basically like a projection of points in multidimensional space to eliminate one dimension, then another, then another. For example, starting in 3D space, we would project the points onto some plane cutting through that space, and do it in such a way as to minimize error. But that would have to be done consistently with all of your fMRI data. If you eliminate a dimension, it has to be done consistently for the entire data set. Do you have an intuition from the study of neuroscience as to what that "immaterial" dimension might be?

The next hard part is going to be the training. I think you'll need a rather large data set of fMRI images that have been hand-labeled with a word that describes what the subject brain was thinking about at the time the image was taken. That might be expensive to generate, unless labs already working on this problem are sharing heir data.

Hi. Thank you so much for this comment! Im in the bus atm and I won't be able to write a proper response until tomorrow, but I wanted to acknowledge your comment. :)
I'm in the process of writing a post oulining the motivation for this project and the next steps I plan to take.
You are absolutely correct about the low resolution of fMRI imaging. Exciting new techniques like calcium imaging provide much much higher resolution, both temporal and spacial. Most commonly neural decoding is performed using "traditional" ML techniques, and I'm sure I don't need to tell you how promising DL seems for achieving better results. These two developments are the reason I believe a breakthrough in neural decoding is more or less imminent.
And now regarding your concerns for collecting data. In image decoding, a label for a dataset consists of the video the subject was watching while the data was being collected. An example of such a dataset and how to work with it is at http://nilearn.github.io/auto_examples/02_decoding/plot_miyawaki_reconstruction.html. The data I plan on using comes from mice.
All right, this is my stop. I hope Ive answered some of your questions. :))

Incidentally, there's a research group in China that's working on this too. And there are probably others. I just remember reading about the China group in MIT Technology Review. https://www.technologyreview.com/s/604332/mind-reading-algorithms-reconstruct-what-youre-seeing-using-brain-scan-data

Thanks, Ill check it out!