The rapid development of information technology has fueled an ever-increasing demand for ultrafast and ultralow-energy-consumption computing. Existing computing instruments are pre-dominantly electronic processors. The scaling of computing speed is limited not only by data transfer between memory and processing units, but also by RC delay associated with integrated circuits. Using photons as information carriers is a promising alternative. Here, we report a strategy to realize ultrafast and ultralow-energy-consumption all-optical computing based on convolutional neural networks, leveraging entirely linear optical interactions. The device is constructed from cascaded silicon Y-shaped waveguides with side-coupled silicon waveguide segments to enable complete phase and amplitude control in each waveguide branch. The generic device concept can be used for equation solving, multifunctional logic operation, Fourier transformation, series expanding and encoding, as well as many other mathematical operations. Multiple computing functions were experimentally demonstrated to validate all-optical computing performances. The time-of-flight of light through the network structure corresponds to an ultrafast computing time of the order of several picoseconds with an ultralow energy consumption of dozens of femtojoules per bit. Our approach can be further expanded to fulfill other complex computing tasks based on non-von Neumann architectures and thus paves a new way for on-chip all-optical computing.