In 1988, at Bell Labs, a young Yann LeCun sat in front of a computer, excitedly demonstrating LeNet, the very first Convolutional Neural Network (CNN) he had built. It successfully recognized handwritten ZIP codes on envelopes, marking the first time CNNs revealed their immense practical potential. Although LeNet was not widely adopted by the academic community at the time—let alone applied to other fields—the gears of fate had already begun to turn quietly…

Fast forward to 2012. Coming seemingly out of nowhere, AlexNet slashed the image recognition error rate to 15.3% in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC-2012)—a 10.8% reduction compared to the previous best model (26.1%). With this triumph, deep learning upended traditional machine learning methods. In a later interview, Fei-Fei Li asked Yann LeCun: “Yann, was this model fundamentally different from the model (LeNet) you invented in the ’80s?” LeCun replied: “No, it’s exactly the same, and it won all the competitions!” The seeds planted by LeNet in 1988 had finally broken through the soil. After years of dormancy, excitement in the AI field surged overnight, opening a brand-new chapter for neural networks.

Artificial Intelligence: A Journey Toward Dawn Through the Winters#

Artificial Intelligence (AI), also known as machine intelligence, traces its conceptual roots back to the theory of computation and the “Turing Machine” model proposed by the British mathematician and logician Alan Turing in 1936. In his seminal 1950 paper Computing Machinery and Intelligence, Turing introduced the “Turing Test” to further explore whether machines could exhibit human-like intelligence, mimic human cognitive and learning processes, and solve problems using logical reasoning and given information. In 1956, John McCarthy and his colleagues formally coined the term “Artificial Intelligence” at the Dartmouth Workshop and set out to study how machines could simulate intelligent human behavior—marking the official dawn of the AI era.

Throughout the history of AI, the fundamental question of “what constitutes intelligence” has remained at the center of academic debate, giving rise to two major schools of thought: Symbolicism and Connectionism. The former holds that the essence of intelligence lies in symbols and rules, whereas the latter argues that the key to intelligence lies in the strength of connections between neurons. The collision and interplay between these two paradigms have shaped the turbulent nearly 70-year journey of AI.

The Era of Symbolicism#

In studying artificial intelligence, the Symbolicist school focuses on enabling machines to achieve human-like cognitive functions by manipulating symbols—encoded information such as text, images, and sounds. Heavily inspired by rationalist philosophy, symbolicists view intelligence as computation and reasoning within a world of symbols. Consider the classic syllogism: Socrates is a man; all men are mortal; therefore, Socrates is mortal. Operating through a specific symbolic language, a machine can always deduce valid conclusions from true premises. Imagine the Symbolicist school teaching a child mathematics: they teach the child that the “+” symbol represents addition, and then how to compute by combining numbers with the “+” symbol (e.g., 3 + 2 = ?). In this process, intelligence does not stem from mimicking the mental activities of an adult performing addition, nor does it concern how the biological brain processes mathematical information; rather, it is manifested through understanding and applying symbolic rules.

In 1956, Allen Newell and Herbert Simon developed the “Logic Theorist,” the first program capable of proving mathematical logic theorems. By applying a set of predefined rules for deductive reasoning, it successfully proved numerous propositions in Chapter 2 of Bertrand Russell’s Principia Mathematica, marking a major initial triumph for the Symbolicist school in both theory and practice. Shortly thereafter, in 1958, John McCarthy invented the Lisp programming language and proposed the world’s first complete AI system—a hypothetical program capable of learning like a human—providing AI researchers with a unified programming environment. From then on, symbolic AI surged ahead at full steam. This era gave birth to milestone applications such as the industrial robot Unimate, the “psychotherapist” ELIZA (a text-based precursor to Siri), and SHRDLU, which could execute simple commands in a virtual block world. By this time, techniques for formalizing linguistic text into computer programs had matured significantly. Encoding vast amounts of domain expertise and inference rules gave rise to a series of expert systems—such as MYCIN, a medical diagnosis system for bacterial infections whose knowledge base contained a massive collection of diagnostic rules distilled from interviews with medical experts.

The Two AI Winters#

Imagine trying to build a medical diagnostic expert system under the symbolic paradigm: you would have to manually code thousands of intricate pathological and diagnostic rules. Yet medical knowledge is constantly expanding and evolving. Because these systems could not learn directly from experience—much like a student who relies purely on rote memorization—researchers had to endlessly expand and revise the rule base, a task that quickly grew unmanageably complex. Coupled with limited computing hardware, symbolic systems fell far short of practical demands in both diagnostic accuracy and efficiency when confronted with complex diseases—at a time when the public expected AI to rival or even surpass human doctors in delivering instant, accurate diagnoses. People began to realize that AI was failing to deliver tangible productive value to society. In 1970, the U.S. Department of Defense began slashing budgets for basic research in AI. Three years later, Sir James Lighthill revealed in his Artificial Intelligence: A General Survey (the Lighthill Report) that many ongoing AI projects were falling far short of expectations and that early investments had yielded little return. Consequently, the British government refused to pour any more money into what seemed like a bottomless pit. With funding cut off, AI development ground to a halt, ushering in the First AI Winter.

In the 1980s, the wheels of AI began to turn once more. In 1981, Japan launched its ambitious Fifth Generation Computer Systems project, aiming to build highly intelligent computer systems—a move that sparked a new global wave of enthusiasm for AI. With the advent and proliferation of microprocessors, computing and data processing speeds had improved dramatically compared to the 1970s, enabling certain expert systems to achieve commercial deployment and generate significant economic returns. For instance, Ford Motor Company utilized the R1 expert system for automotive manufacturing diagnostics, helping to reduce production line downtime. However, the fundamental technical bottleneck of expert systems—their heavy reliance on human experts to manually encode rules of knowledge—remained unresolved. This made it difficult for them to handle ambiguity, uncertainty, and complex environmental changes, leaving their real-world performance far below the lofty expectations of the public and investors. The economic crash of 1987 severely shook investor confidence in high-risk AI technologies, driving numerous computer companies into bankruptcy. Guided by Symbolicism, the AI industry helplessly slid into its Second Winter.

The Crucible of Connectionism#

Imagine you have a pile of colorful building blocks, each capable of receiving and passing along information. Stacked layer upon layer, the blocks at the very bottom take in raw input (such as an image), and each subsequent layer processes the information according to certain rules before passing it upward. This “team of blocks” possesses a special ability—learning. Much like how we remember the right answers after doing exercises and correcting our mistakes, these blocks can learn to tackle complex tasks like recognizing images and understanding language through continuous trial and adjustment. This is the core idea behind Connectionism, which can be seen to some extent as the embodiment of empiricist philosophy in AI (i.e., that knowledge derives from experience). By this point, you may have noticed that right up until AI entered its Second Winter, Symbolicism seemed to hold the upper hand in the rivalry between the two schools. What did the Connectionist school go through during this period? And how did it turn the tide against all odds to bring a new dawn to AI?

Deeper Networks#

In 1943, American mathematician Warren McCulloch and logician Walter Pitts proposed the first biologically inspired artificial neuron model (named the M-P model after their initials) in their paper A Logical Calculus of the Ideas Immanent in Nervous Activity, marking the birth of the Connectionist school. By 1958, Frank Rosenblatt designed the Perceptron, the first physically constructed single-layer neural network (akin to having just a single layer of building blocks). While early Perceptrons excelled at solving linearly separable problems, they hit a wall when dealing with non-linear tasks. Crucially, the power of a neural network depends heavily on its depth, which involves adjusting weights and learning across vast numbers of interconnected neurons. Given the computing constraints of the era, training large-scale neural networks was exceedingly difficult. Moreover, effectively training neural networks requires massive amounts of labeled data, and at a time when data collection and storage were prohibitively expensive, neural networks performed poorly on many real-world tasks. In 1974, Paul Werbos first proposed the Backpropagation (BP) algorithm in his doctoral dissertation—a pivotal technique for training multi-layer (large-scale) neural networks. However, it received little attention at the time, as the AI field was in the depths of its First Winter. Furthermore, compared to Symbolicism, which emphasized explicit logic and clear rules, a neural network operated like a “black box.” Its opaque learning mechanism and uninterpretable results made it difficult for the mainstream scientific community to accept.

The first turning point arrived in 1986, when Geoffrey Hinton—one of the three “Godfathers of Deep Learning” (alongside Yoshua Bengio and Yann LeCun)—published a groundbreaking work on the Backpropagation algorithm in Parallel Distributed Processing. It sent shockwaves through the field, prompting researchers to begin using neural networks for feature extraction, pattern matching, and solving problems in image processing and computer vision. Unfortunately, the previously overhyped AI bubble had already burst, leaving governments and entrepreneurs deeply cautious about investing in the AI industry. After seven years of iteration on the LeNet model mentioned at the beginning of this chapter, LeNet-5 became the first commercial convolutional neural network in 1995, successfully reading 10% to 20% of all checks issued in the United States—one of the most remarkable achievements of neural networks at the time. Yet LeNet-5 failed to win over venture capital. Researchers and investors of that era favored machine learning methods grounded in deterministic mathematical theorems, such as Support Vector Machines (SVMs) and Boosting. Because a complex, hard-to-explain technology often implied high risk, non-mainstream neural networks naturally kept capital at bay.

Breaking Through Prejudice: The Triumph of Connectionism#

Entering the early 21st century, neural networks found themselves in dire straits. Even though they worked well in practice, most researchers dismissed neural network research as a dead end because no one at the time could explain why they worked so well. Left with little choice, Yann LeCun and his peers rebranded neural networks under a new guise—“Deep Learning”—reintroducing them to the public with a fresh identity. Today, deep learning is white-hot, yet even now its “black box” remains largely unopened. What, then, convinced the once-“stubborn” scholars of the past to set aside their prejudice against deep learning? Perhaps, in the face of overwhelming gains in task accuracy, the opacity of the “black box” temporarily ceased to matter.

The two wings that enabled deep learning to soar into the skies were data and computing power. Breakthroughs in GPUs multiplied computational capacity exponentially, and around 2006, researchers successfully harnessed GPUs to accelerate neural network computations. For deep learning, the GPU was like a high-horsepower engine—you could almost hear it roaring, ready to take off. And if data is the oil of the new era, before Fei-Fei Li released ImageNet in 2009, it was still just dinosaur bones buried underground. Fei-Fei Li firmly believed that “data will redefine how we think about models.” Facts proved that data is indeed precious fuel: it allows algorithms to perceive a more authentic and complex world, achieving what we now call “generalization.” When GPUs met big data, neural network researchers realized their moment had finally arrived. As noted at the beginning of this chapter, 2012 was the decisive year: Geoffrey Hinton’s team swept the ILSVRC-2012 competition with AlexNet.

Technological progress never happens overnight; it is the fruit of repeated experimentation, failure, incremental accumulation, and exploration. Neural networks—and the steadfast researchers behind them—walked a long road to get here. The refinement and optimization of foundational theories and techniques forged more mature deep learning methods, which, combined with larger datasets and greater computing power, propelled a new leap forward in AI research.

The Wheel of History Never Stops Turning#

In humanity’s relentless pursuit of machine intelligence, the debate between Connectionism and Symbolicism actually mirrors a deeper clash and convergence within the philosophy of science regarding how to model complex systems. As these two schools of thought intertwined through the long history of AI, their journey was never a simple zero-sum contest. While Connectionism relies on data-driven learning, network architecture design and feature modeling still depend on the guidance of prior knowledge. Conversely, while Symbolicism emphasizes logical reasoning, it also leverages machine learning algorithms to learn and refine knowledge from data. The iterative leap from LeNet to AlexNet and now to ChatGPT-5 bears witness to how technological progress harnesses the dividends of its era—massive datasets, formidable computing power, and more forward-looking vision—to break through existing boundaries, continually optimizing and infusing innovative elements into what is fundamentally the same neural network paradigm, transforming it into a powerful engine driving epochal change.

Every major breakthrough in artificial intelligence is accompanied by profound reflection on the boundaries of future cognition and the very nature of intelligence. Human understanding of technology is inevitably bound by the limitations of its time. As Fei-Fei Li once remarked, “How we are remembered depends on what we do right now.” Perhaps, in an era of rapid technological refinement, we ought to think more carefully about the emerging landscape of human-machine relations. Beneath the current fervor lies a much more open-ended question: within our existing technological and theoretical frameworks, will artificial intelligence touch upon and ultimately transcend the deep structures of the human mind, or will it, in the face of new challenges, force us to re-examine and redefine “intelligence” itself?

Core Technologies of Connectionism#

At the close of this section, we provide a brief introduction to the classic techniques of Connectionism, which will help readers navigate the subsequent chapters of this book.

1. The Perceptron and Multi-Layer Neural Networks (Deep Learning Networks)#

The Perceptron is the most foundational model designed to mimic neurons in the human brain. It takes in multiple pieces of information (much like a child viewing building blocks of various colors from different angles), where each piece carries a specific weight (the intensity of each block’s color), and sums them up (putting all the blocks together to see the overall color). If the combined result reaches a certain threshold (for instance, predominantly dark blocks indicating the color red), the Perceptron makes a decision (“I see red”). By continuously adjusting the importance (or weight) of each block, the Perceptron learns to distinguish between different inputs. Just like stacking building blocks, multiple Perceptrons can be layered on top of one another, with each layer passing its processed results to the next. This makes it possible to tackle much more complex tasks, such as distinguishing between photos of different animals. Each layer of Perceptrons learns to recognize specific features (such as eyes, ears, etc.), ultimately combining them into a comprehensive understanding of the entire image.

2. The Backpropagation Algorithm#

Backpropagation is like teaching an intricate building-block robot how to complete a specific task. Suppose we ask it to assemble a pattern out of blocks. If it gets the pattern wrong, we need to tell it where the mistake lies. However, because there are so many blocks and the structure is so complex, pointing out exactly which block needs to move—and by how much—is no easy feat. Backpropagation solves this by inspecting the final assembled result first, and then tracing backward step by step through each layer of blocks to figure out how fine-tuning each individual block can bring the outcome closer to the correct answer. In this way, after every attempt, the robot adjusts the positions of its blocks based on this feedback. Through repeated training and correction, it eventually learns to assemble a wide variety of patterns with pinpoint accuracy. In deep learning, Backpropagation is the crucial mechanism that helps a neural network progressively improve and find its optimal parameter settings, enabling the network to perform tasks ever more effectively.

3. Two Powerhouses for Sequential and Image Data: RNNs and CNNs#

Recurrent Neural Network (RNN)

An RNN is like a storytelling robot with a memory. Every time it hears a word, it jots it down in memory and uses what it remembered from the previous step to guess the next word. This kind of “memory” machine, capable of processing information that unfolds in chronological order, is a Recurrent Neural Network. When processing sentences or audio, it works much like we do when reading a story—predicting what comes next based on what came before. However, this storytelling robot sometimes forgets important details from much earlier in the narrative.

Convolutional Neural Network (CNN)

A CNN is an intelligent jigsaw-puzzle expert. Using a sliding window, it scans an image and extracts a variety of small local patterns, such as edges and textures. By stacking multiple convolutional layers, it achieves higher levels of abstraction—for example, piecing together identified eyes and a nose to determine whether the image shows a puppy or a bunny. Unlike an RNN, where the output at each time step depends on both current and past information, the spatial relationships between pixels in an image are fixed rather than arranged in chronological sequence. Through local convolution and pooling operations across neighboring pixels, a CNN can efficiently capture the spatial structure and local invariance of an image, making it far better suited for image processing tasks.

4. The Butterfly That Flapped Its Wings: Transformer — Attention Is All You Need#

As the core algorithm behind ChatGPT, the Transformer unleashed a technological storm across the AI industry, fundamentally revolutionizing how we process sequential data—especially natural language. If you ask an RNN and a Transformer to read a thick book at the same time, the RNN will crawl along like a snail, reading word by word. Although it tries hard to remember what it read earlier, it still tends to forget key words from chapters long past. The Transformer, by contrast, possesses magical X-ray vision (the “Self-Attention” mechanism). It can take in the entire story at a single glance, simultaneously grasping the meanings of numerous words and sentences while identifying the connections between them, regardless of how long the story is.

It is like building a towering castle out of blocks: whereas we previously had to place the blocks one by one, the Transformer allows us to lay down many blocks all at once, erecting a tall and sturdy castle in no time. Leveraging its mastery of “parallel computing,” the Transformer enables computers to process vast amounts of information simultaneously rather than waiting in line. This has empowered scientists to construct massive models composed of billions of blocks (parameters). Much like real-life Transformers, these models have grown ever more powerful and capable, ready to tackle a wide array of linguistic challenges.

5. Reinforcement Learning#

While deep learning learns to accurately recognize or predict things from massive amounts of labeled data, Reinforcement Learning (RL) does not require you to provide the expected answers upfront during training; instead, it learns from feedback on whether the outcome of its actions was good or bad. Much like a child playing a video game, every move yields immediate feedback from the game (a score reward or a penalty). Through continuous trial and error, the child learns which actions lead to favorable outcomes and thus achieves a higher score. In reinforcement learning, RNNs and CNNs are commonly used to process the various forms of state information received by an agent, before passing that processed information on to a policy network or value network for action selection and learning. In AlphaGo, for instance, CNNs were used to evaluate how strongly the current board configuration influenced the ultimate outcome of the game—that is, predicting the probability of Black or White winning given a specific board layout.

Author Information & References#

Fei Huang
Ph.D. Candidate (Class of 2022), Department of Urban Spatial Information Engineering, Shenzhen University.
Research Interests: GeoAI, Geospatial Tokenization, Human Mobility Generation, Geospatial Data Representation Learning.

This chapter was written by Fei Huang with the assistance of AI tools (ChatGPT-4, Tongyi Qianwen, Jimeng AI, etc.). Illustrations used in the text are sourced from the Internet.

The cover image was generated by Jimeng AI with the prompt: A dawn scene in deep winter, a tall robot walking hand in hand with a human across a frozen river, their backs facing the golden morning light in the distance, footprints in the snow stretching toward misty snow-capped mountains, intricate frost details, abstract cinematic composition, rich texture, frosted glass effect.

References#

1. https://www.storm.mg/article/1375326
2. https://www.jiqizhixin.com/articles/2018-12-11-20
3. https://m.thepaper.cn/baijiahao_19930583
4. https://www.aminer.cn/ai-history
5. https://zhuanlan.zhihu.com/p/28142670
6. LeCun, Y. 2021. Quand la machine apprend (The Road to Science / 科学之路). Beijing: CITIC Press Group.
7. https://www.youtube.com/watch?v=97qFVWjtbxE