A New Approach to Computation Reimagines Artificial Intelligence: Hyperdimensional Computing (quantamagazine.org) 43
Quanta magazine thinks there's a better alternative to the artificial neural networks (or ANNs) powering AI systems. (Alternate URL)
For one, ANNs are "super power-hungry," said Cornelia Fermüller, a computer scientist at the University of Maryland. "And the other issue is [their] lack of transparency." Such systems are so complicated that no one truly understands what they're doing, or why they work so well. This, in turn, makes it almost impossible to get them to reason by analogy, which is what humans do — using symbols for objects, ideas and the relationships between them....
Bruno Olshausen, a neuroscientist at the University of California, Berkeley, and others argue that information in the brain is represented by the activity of numerous neurons... This is the starting point for a radically different approach to computation known as hyperdimensional computing. The key is that each piece of information, such as the notion of a car, or its make, model or color, or all of it together, is represented as a single entity: a hyperdimensional vector. A vector is simply an ordered array of numbers. A 3D vector, for example, comprises three numbers: the x, y and z coordinates of a point in 3D space. A hyperdimensional vector, or hypervector, could be an array of 10,000 numbers, say, representing a point in 10,000-dimensional space. These mathematical objects and the algebra to manipulate them are flexible and powerful enough to take modern computing beyond some of its current limitations and foster a new approach to artificial intelligence...
Hyperdimensional computing tolerates errors better, because even if a hypervector suffers significant numbers of random bit flips, it is still close to the original vector. This implies that any reasoning using these vectors is not meaningfully impacted in the face of errors. The team of Xun Jiao, a computer scientist at Villanova University, has shown that these systems are at least 10 times more tolerant of hardware faults than traditional ANNs, which themselves are orders of magnitude more resilient than traditional computing architectures...
All of these benefits over traditional computing suggest that hyperdimensional computing is well suited for a new generation of extremely sturdy, low-power hardware. It's also compatible with "in-memory computing systems," which perform the computing on the same hardware that stores data (unlike existing von Neumann computers that inefficiently shuttle data between memory and the central processing unit). Some of these new devices can be analog, operating at very low voltages, making them energy-efficient but also prone to random noise.
Thanks to Slashdot reader ZipNada for sharing the article.
Bruno Olshausen, a neuroscientist at the University of California, Berkeley, and others argue that information in the brain is represented by the activity of numerous neurons... This is the starting point for a radically different approach to computation known as hyperdimensional computing. The key is that each piece of information, such as the notion of a car, or its make, model or color, or all of it together, is represented as a single entity: a hyperdimensional vector. A vector is simply an ordered array of numbers. A 3D vector, for example, comprises three numbers: the x, y and z coordinates of a point in 3D space. A hyperdimensional vector, or hypervector, could be an array of 10,000 numbers, say, representing a point in 10,000-dimensional space. These mathematical objects and the algebra to manipulate them are flexible and powerful enough to take modern computing beyond some of its current limitations and foster a new approach to artificial intelligence...
Hyperdimensional computing tolerates errors better, because even if a hypervector suffers significant numbers of random bit flips, it is still close to the original vector. This implies that any reasoning using these vectors is not meaningfully impacted in the face of errors. The team of Xun Jiao, a computer scientist at Villanova University, has shown that these systems are at least 10 times more tolerant of hardware faults than traditional ANNs, which themselves are orders of magnitude more resilient than traditional computing architectures...
All of these benefits over traditional computing suggest that hyperdimensional computing is well suited for a new generation of extremely sturdy, low-power hardware. It's also compatible with "in-memory computing systems," which perform the computing on the same hardware that stores data (unlike existing von Neumann computers that inefficiently shuttle data between memory and the central processing unit). Some of these new devices can be analog, operating at very low voltages, making them energy-efficient but also prone to random noise.
Thanks to Slashdot reader ZipNada for sharing the article.
Hyperdimensional vector (Score:4, Interesting)
"A hyperdimensional vector, or hypervector, could be an array of 10,000 numbers, say, representing a point in 10,000-dimensional space."
Also known as a perl hash.
Re: Hyperdimensional vector (Score:1)
Also know as vector.
Re: (Score:2)
challenge.
which is more efficient.
a neural network
or the hyper d.
let bench marks be the center referee
Quantum carburetor (Score:1)
Quantum carburetor? Jesus, Morty. You canâ(TM)t just add a sci-fi word to a car word and hope it means something. Huh, looks like somethingâ(TM)s wrong with the microverse battery.
Well I'll see your hyperdimensional computing and raise you a fermion superfluidity matrix.
Word Embeddings (Score:4, Insightful)
Re: (Score:3)
Re:Word Embeddings (Score:5, Informative)
This sounds a lot like what most AI systems already do with their word embedding.
Vector Space Models are nothing new.
Multidimensional representations of various kinds have been the basis of a lot of classic AI, going back to the late 1970s. Frames, K-Lines, Feature Vectors, and so on. The challenge is the specific representation with respect to computational limits. And the early AI processors (such as Hillis's original Connection Machine) were highly parallel machines: in-memory nodes connected by hyperdimensional communication networks. And analogical reasoning was one of the basic approaches (Winston, Mallery, et al). So in that sense, none of this stuff is new.
The trick is (as always) what is being represented and what operations are available to manipulate that. The current NN/ML systems represent very primitive things -- not, for example, "facts". That includes the LLMs which represent something rather dubious: symbols without any meaning (or any way to really attach any meaning).
From TFS, it sounds like someone wants to do some real AI for a change, and is looking at the computational power available for more realistic "neurons", and the old basics of actual knowledge representation. The logic/analogy/inference operations would be on top of the hyperdimensional vectors. So this is interesting, and quite beyond what current NNs are doing.
At least I hope that's what they're talking about; I just read TFS.
Re:Word Embeddings (Score:4, Informative)
Word embedding implies far less dimensions (Score:4, Interesting)
Re:Word embedding implies far less dimensions (Score:4, Interesting)
HYPErdimensional computing (Score:2)
magic box is broke (Score:3)
It could be that we need to take a step backwards before the next step forward.
Mug sees unicorn among hypedust maelstrom (Score:2)
Everybody board the hype train! (Score:2)
I counted at least 6 buzzwords to justify the reasoning.
Processor in memory (Score:3)
Sadly have been hearing about PIM for decades and very little has ever seemed to materialize. Closest I know of is Mythic AI's analog processor.. which for some reason I cannot understand is operating on a shoestring budget and not having billions of dollars thrown at it.
An interesting comparison between HDC and NN:
https://arxiv.org/pdf/2207.129... [arxiv.org]
Re: (Score:2)
FWIW, you can represent ANNs as vectors (Score:2)
Represent each input and neuron by its outgoing weights, and you get vectors.
Or for all the outgoing weights for a layer, represent it with a matrix.
Before we had hardware acceleration I got a lot of mileage out of BLAS when training ANNs.
It's still an LLM (Score:1)
Where’s the beef? (Score:2)
I distinctly remember hearing about this form of representation for concepts in an undergraduate seminar class I took, but the use case wasn’t AI back then, it was translation. More or less, words in different languages that have identical meanings would live at the same hyper dimensional point, and the closer a non-identical value was to another, the closer their meanings.
Here we are, 20 years later. I haven’t exactly heard about this work taking the world by storm, which gives me the impressio
Re: (Score:2)
forest for the trees (Score:2)
No one truly understands what they're doing? (Score:3)
no one truly understands what they're doing
This is false. Just because the *author* doesn't understand how LLMs work, doesn't mean that *no one* does.
If no one understood, then LLMs wouldn't work, and it wouldn't be possible for competitors to spring up with their own versions. The people who develop LLMs not only understand how they work, but they are able to tweak and mold the output based on requirements. For example, it won't take long before LLM designers figure out how to incorporate advertising into their systems, because they're not going to keep offering their systems to people for free. And as they discover disturbing types of patterns (such as relating to extremist political ideologies) the designers will find ways to exclude those topic areas.
The average layperson doesn't understand how a computer works, but many of us here on slashdot understand computers and software at varying levels. Why wouldn't LLMs have the same pattern of being well-understood by some, and not at all by others?
Re: (Score:3)
This is false. Just because the *author* doesn't understand how LLMs work, doesn't mean that *no one* does.
It's also true just because *your* statement is correct doesn't necessarily mean *anyone* understands either.
Understanding isn't required to make use of something. People for example have developed drugs without understanding why they work. Some industrial annealing processes exploit chance discoveries of effects that are exploited without underlying explanation.
If no one understood, then LLMs wouldn't work, and it wouldn't be possible for competitors to spring up with their own versions. The people who develop LLMs not only understand how they work, but they are able to tweak and mold the output based on requirements. For example, it won't take long before LLM designers figure out how to incorporate advertising into their systems, because they're not going to keep offering their systems to people for free. And as they discover disturbing types of patterns (such as relating to extremist political ideologies) the designers will find ways to exclude those topic areas.
People understand enough to train models and execute them yet training process is based on learned experience and trial and error rather than und
Re: (Score:2)
You seem to be disagreeing with my statement, but none of your points actually contradict my statement, as far as I can see.
No one understands ALL of anything. They understand enough to make use of the things they need to make use of. But that doesn't mean no one understands things, including LLMs. Somebody built it, somebody maintains it, somebody enhances it, somebody trains it. Each of those people understand enough to do their jobs. Parts of the system are a black box to some of those people, but not al
Re: (Score:2)
No one understands ALL of anything. They understand enough to make use of the things they need to make use of. But that doesn't mean no one understands things, including LLMs. Somebody built it, somebody maintains it, somebody enhances it, somebody trains it. Each of those people understand enough to do their jobs. Parts of the system are a black box to some of those people, but not all.
The quote is talking explicitly about the underlying trained neural network.
Somebody built it, somebody maintains it, somebody enhances it, somebody trains it. Each of those people understand enough to do their jobs. Parts of the system are a black box to some of those people, but not all.
This is not a problem of compartmentalization. It's a problem of being able to understand structure of trained neural models. You can communicate with the model through interfaces yet understanding what it is doing to arrive at an output is largely magic to all.
Re: (Score:2)
I do realize we're talking about the underlying neural network. I've personally built and trained a neural network, back in the 90s. They aren't so mysterious once you start to see how they actually work. People tend to superimpose mystery and human intelligence on what is really, at it's core, a simple structure and process. At it's core, it's really about pattern recognition and using those patterns to make predictions.
Context error (Score:2)
You've misunderstood the context. It's not at all about understanding about how neural networks are constructed, trained, or queried; it's about understanding the encoded information structure within any particular multi-billion parameter network after they have been trained such that one can know what token sequence they will produce for any given query
Re: (Score:2)
It certainly is possible to debug LLMs.
https://blog.allenai.org/intro... [allenai.org]
A lot of traditional code was impossible or difficult to debug because it was written procedurally, leading to a lot of spaghetti. As the art and science of programming developed, we found ways to reorganize our code in ways that could be empirically debugged. LLMs are no different. An inability to debug them indicates either 1) the programmer lacks the skill or 2) the tool was written in a way so as to make debugging unnecessarily diff
Hyperdimensional? Oh brother... (Score:2)
Sounds like hype to me.
How is this different from standard AI? (Score:2)
So you represent your data points as "hyperdimensional" vectors. They're still data points, and the patterns that emerge from them can still be analyzed through traditional AI models. AI doesn't care how a data point is represented, it just looks for patterns.
trivial (Score:2)
This seems to be completely trivial. Essentially all learning programs represent data as vectors in a very high dimensional space. A single 4k color image is a vector in a 24 million dimensional space. Neural nets represent intermediate data in a space of as many dimensions as there are "neurons." Changing bits is irrelevant in the face of efficient error-correcting codes. Etc. etc. Seems trivial. I have not read more than the summary, so maybe the summary is just stupid.
Hyperdimensional Computing old new :) (Score:1)
I don't get it (Score:2)
Changing the virtual model doesn't change the hardware - and neural nets using clusters of neurons with low triggering thresholds are supposed to be pretty efficient, and it's been a while since IBM said they'd build the first 'memristor' that would enable building neural nets in hardware with higher density and lower power use than transistors.
Did memristors fail for some reason, or are they just coming along a lot more slowly than anticipated?
Hyperdimension Neptunia (Score:2)
Re: Hyperdimension Neptunia (Score:1)
Is this one of those AI generated articles? I'm reading this nonsense and part of me is like "holy crap the technology advanced so far, I'm so far behind that I don't understand the shit they are saying. "
Re: (Score:2)