Another Eureka moment for AI?

What happened?

• A paper was published by MIT rethinking Artificial Neural Networks, the fundamental technology behind all recent AI breakthroughs: Kolmogorov-Arnold Networks
• Many researchers believe this is another tipping point and will open up new possibilities.

Why does it matter?

• Potentially touching the heart of AI and bringing it to a new level of accuracy and efficiency, while also making AI more understandable.
• Making existing applications more efficient and insightful.
• Creates new opportunities for applications.

What to do?

• Decisionmakers without direct involvement in AI development: Monitor the evolution and watch out for applications in your industry.
• Decisionmakers with direct involvement in AI development: Look for potential in applications mainly focussed on classification and regression applications. Keep an eye out for implementations in other architectures.

What’s next?

• The paper itself already contains experiments that outperform current state-of-the-art benchmarks. New citations are being published almost on a daily basis and applications are being developed in image processing, sound and speech recognition, financial and predictive modelling and medical applications.

However…

• The paper was published beginning of May 2024, so the type of network is extremely new. The theoretical potential is significant, but to be validated in practice.
• Currently, the networks are still quite complex and require a lot of finetuning and prior knowledge. Automated optimization techniques are already proposed, but need to be extended.

The tech summary

For deep-tech executives

• Fixed activation functions in traditional neural networks are replaced by learnable functions.
• Implemented by univariate splines on the edges of a multilayer perceptron.
• The network can be automatically pruned to remove irrelevant branches, resulting in smaller networks.
• Networks can be optimized without the need for complete retraining. Splines can be made more fine-grained and individually reshaped to their original coarse-grained shape.

The Fineprint

Artificial Neural Networks (ANN) are at the core of nearly all modern AI technologies. They attempt to simulate how a biological brain works by connecting many brain cells (neurons) with each other. ANNs work because they can capture non-linear relationships within data. This non-linearity is enforced through the use of activation functions applied to the output of each neuron. However, the type of these activation functions are manually selected and imposed on the architecture of the network, which makes them assumptions or even arbitrary choices.

Kolmogorov-Arnold Networks, on the other hand, leverage a long-standing mathematical theorem to use splines (chains of polynomial functions) to approximate complex relationships and to enable their non-linearity. The combination of a classical multilayer perceptron and splines, results in a network that can learn features as normal ANNs, but also finetune them in a straight-forward and interpretable manner.

Comparison of traditional MLPs and KANs [Source].