An autonomous laser kirigami method with low-cost real-time vision-based surface deformation feedback system

We introduce an autonomous laser kirigami technique, a novel custom manufacturing machine system which functions somewhat similar to a photocopier. This technique is capable of creating functional freeform shell structures using cutting and folding (kirigami) operations on sheet precursors. Conventional laser kirigami techniques are operated manually and rely heavily on precise calibrations. However, it is unrealistic to design and plan out the process (open loop) to realize arbitrary geometric features from a wide variety of materials. In our work, we develop and demonstrate a completely autonomous system, which is composed of a laser system, a 4-axis robotic arm, a real-time vision-based surface deformation monitoring system, and an associated control system. The laser system is based on the Lasersaur, which is a 120-Watt CO2 open-source laser cutter. The robotic arm is employed to precisely adjust the distance between a workpiece and the laser lens so that a focused and defocused laser beam can be used to cut and fold the workpiece respectively. The four-axis robotic arm provides flexibility for expanding the limits of possible shapes, compared to conventional laser machine setups where the workpiece is fixed on rigid holders. The real-time vision-based surface deformation monitoring system is composed of four low-cost cameras, an integrated AI-assisted algorithm, and the sensors (detachable planar markers) mounted on the polymer-based sheet precursors, and allows real-time monitoring of the sheet forming process with a geometric feature estimation error less than 5% and delay time around 100 ms. The developed control system manages the laser power, the laser scanning speed, the motion of the robotic arm based on the designed plan, and the close-loop feedback provided by the vision-based surface deformation monitoring system. This cyber-physical kirigami platform can operate a sequence of cutting and folding processes in order to create kirigami objects. Hence, complicated kirigami design products with various different polygonal structures can be realized by undergoing sequential designed laser cuts and bends (at any folding angles within designed geometric tolerance) using this autonomous kirigami platform.


Introduction
Recent advances in cyberphysical systems (CPS) methodologies and manufacturing technologies are fueling the demand for custom manufacturing of high-quality products at a mass production cost structure, especially in medical, energy and aerospace industries. Breakthroughs at the nexus of computation-communication-control [1, 2] for quality Zimo Wang zimowang@binghamton.edu Extended author information available on the last page of the article. assurance, along with the lines of what the photocopying sector has achieved, are necessary to deploy custom manufacturing technologies for free-form functional components as service kiosks [3,4].
Currently, custom manufacturing has been primarily based on 3D printing or additive manufacturing (AM) using wire or powder precursors. However, the additive manufacturing processes have some shortcomings for lowvolume production at a mass production rate structure. First, AM processes commonly require hours to print and are limited to small prototypes. A significant fraction of the products are based on creating hollow or shelllike objects, and the 3D printing is extremely slow and difficult to create such shell-like objects. In addition, powder precursors require additional safety precautions during material handling due to environmental, health, and safety considerations.
A kirigami system offers considerable advantages for custom manufacturing over conventional powder-based 3D printing, because sheet precursors are cheaper and easier to handle, and kirigami process is faster (minutes vs hours) at creating complex parts including large lightweight structurally robust components (e.g., orthotics and furniture) with a variety of prefinished surfaces (e.g., colors, decals, sensors).
Laser cutting of sheet precursors offers a way to quickly create intricate 2D patterns from sheets. As a result, it has found extensive use in a variety of operations from supporting hobbyists to mass manufacturers. It is driven by the speed of the process, together with the ease of handling sheet products. However, most of the laser cutting operations are limited to planar objects. Inspired by the ori/kiri-gami, researchers and engineers have devoted extensive efforts to designing a manufacturing process that creates 3D objects from 2D flat sheet precursors [5,6]. Among these manufacturing processes, laser-based techniques which fold 3D objects from planar sheets have received extensive attention in the early research. Bartkowiak et al. [7] have bent thin-section steel sheets into 3D objects by laser. Laser bending of a low carbon steel tube has been experimentally investigated and simulated through an ABAQUS three-dimensional model by Li and Yao [8]. Gisarioa et al. [9] experimentally analysed and simulated (using ABAQUS Explicit 6.12) the laser shaping 3D metal objects process, and studied the forming of 3D objects by bending of flat metal through the external force-assisted laser origami [10]. Mulay et al. [11] developed a numerical model to predict the bending angles of metal sheets heated by laser beam, and Guo et al. [12] proposed an analytical model that can be used to estimate the bending angles of a metal plate heated by a laser beam under bending loads. Ma et al. [13] developed a method to determine optimal parameters for laser origami using metal sheets.
Additionally, Mueller et al. [14] propose laser origami, of which the three main design elements, the bend, the suspender, and the stretch are utilised to form 3D objects from 2D plastic sheets. The core idea behind this is using a focused laser beam as a cutter to obtain the desired pattern, and a defocused laser as a heater to soften the plastic sheets and thus to deform the sheets by their own weight. Umapathi et al. [15] have developed a new technique named LaserStacker consisting of four key elements: cutting, healing a cut, welding, and releasing a weld, which are achieved by the control of focused and defocused laser beam. This enables 2D plastic sheets to stack together in order to obtain desired 3D objects. Beyer et al. [16] introduced another laser "3D printer"-Platener. In their work, the straight and curved plates are extracted from the 3D object. The laser cutter is then employed to cut these extracted parts, and curved plates are formed by heating and then bending straight plates. The 3D object is obtained by assembling all these 2D parts. However, currently the folding process is largely rooted in an openloop control system, and thus heavily requires repetitive calibrations. Such a bending/folding operation is sensitive to the perturbations of the process parameters (e.g., laser focus, laser power, and laser scanning speed and heating time).
Consequently, the open-loop process creates uncertainties in the geometric of finished 3D objects.
There are many other studies inspired by the origami and the kirigami processes. For instance, an interactive system-Origami Desk [17], has been built to teach users physical and spatial activities. It uses projected video clips to show users how folds should be made, projected animations to directly map instructions onto the users' paper, electric field sensing to detect touch inputs on the desk surface, and swept-frequency sensors to detect the paper's folds. Sterman et al. [18] introduced a design of folding material with embedded electronics, such as LED lights and conductive pads, for making an interactive folding experience. Kinoshita and Watanabe [19] proposed an interactive origami support system, which is able to explain the folding operations clearly according to the procedural states of origami predicted by comparing a silhouette with the shape of origami in a camera image. Olberding et al. [20] developed a new approach of fabricating interactive 3D objects based on a combination of foldable geometries and printed electronics, which enable the 3D objects to have the functions of on-surface input and output as well as shape sensing and actuated shape change. In [21,22], reconfigurable structures such as box and triangular pyramid were achieved by folding a pre-patented 2D sheet, of which the complex creases are achieved by filling soft silicon. Liu et al. [23] demonstrated self-folding of thin temperature sensitive polymer sheets. Black ink patterned on the polymer sheet provides localized absorption of light, which heats the underlying polymer to relax and shrink, causing the out of plane deformation. Balkcorn and Mason [24] built the world's first origami-folding robot, which can fold simple origami. Pique et al. [25] introduced a new technique to generate 3D microstructures by controlling out of plane folding of 2D patterns through a variety of laser sensitive actuators. Nisser et al. [26] developed a laser cutter-based platform for fabricating electromechanical devices and robots.
However, efforts towards realizing arbitrary/free shape geometries are currently stagnant in open-loop operations (plan-out), impeding the process from full automation. Unlike in high-volume manufacturing, real-time planning, adaptation, and control are essential for low-volume custom manufacturing systems. The intelligence comprising all the steps involved in the production process and its continuous monitoring should reside in the real-time service, rather than requiring an upfront costly design process for each product. This issue is similar to what the photocopying industry has dealt with, i.e., how to autonomously monitor the process, dynamically optimize tool motion and process parameters and protect the system in case of a fault (e.g., equivalent of a paper jam) to ensure that the custom part meets specifications. Inspired by the laser origami technique [14], this paper reports an autonomous kirigami platform that addresses the issue at the nexus of CPS and manufacturing quality monitoring.
In this paper, we introduce a vision-based measurement technique to measure the bending angle of the sheets in a non-intervention manner. Using the feedback provided by the vision-based measurement system, a controller unit controls the laser power and laser scanning speed along planned folding paths to give rise to desired bending angles within a acceptable geometric tolerance in real time, which is essential for deploying custom manufacturing systems as a service. A 4-axis robotic arm is employed as a fixture to adjust the position and orientation of workpieces in laser machine coordinates to extend the geometry complexity so that the platform can realize various 3D shapes to meet the requirements of kirigami product design.
The rest of the paper is organized as follows: Section 2 describes the fundamental elements of our laser kirigami system. In Section 3, we introduce the vision-based noncontact deformation measuring technique. The closed-loop control system of the kirigami platform is discussed in Section 4. Section 5 demonstrates the capability of the robotic arm introduced in our platform that allows bending complicated shapes. Section 6 presents the core idea of folding path design. Section 7 demonstrates the capability of the developed cyber-physical platform by autonomously creating an illustrative example of a kirigami designed box. Some concluding remarks are provided in Section 8.

Overview of the autonomous laser kirigami system
The autonomous laser kirigami system consists of three subsystems: a control system (master computer), actuators (robotic arm and laser system), a real-time vision-based monitoring system, and a WiFi Router as infrastructure to establish the communication among the subsystems (see Fig. 1). Conventional manufacturing machines have usually been operated and monitored by onsite supervisors, and there have been difficulties in changing or expanding their system configurations. As a cyber-physical system, each component in the autonomous laser kirigami platform is re-configurable and expandable based on the middleware Fig. 1 The autonomous kirigami system consists of three subsystems: a control system (master computer), actuators (robotic arm and laser system), and a real-time vision-based monitoring system. A WiFi router is employed to establish the communication among the master computer, the laser system, and the vision-based monitoring system through Etherware networking. The master computer can control the laser beam power and the motion of the laser head as well as the robotic arm to conduct the kirigami processes (cutting and folding), with the feedback provided by the vision monitoring system (Ether ware in master computer), which easily enables the management of various kinds of devices and allows dynamic system configuration [27]. Because such a middleware [28,29] enables control over a communication network and computations can be carried out elsewhere with seamless migration [30], each subsystem of the laser kirigami system is physically isolated from others and we can easily add additional devices and operate the whole system based on the streams of sensing information and control commands over a network. Figure 2 shows the configuration of the laser kirigami setup. The laser system is developed based on Lasersaur [31] and includes two sub-systems, a CO 2 laser source and a laser head motion system. The robotic arm is used to hold the workpiece and therefore it can adjust the distance between a workpiece and laser lens (see Fig. 3). When the distance is equal to the laser lens' focal length, the laser beam is well-focused on the region of incidence on the workpiece and cuts the workpiece along the designed cutting path. When the distance is greater than the laser lens' focal length, the laser beam is defocused and heats the workpiece along the designed folding path. As the temperature of the workpiece along a folding path rises and surpasses the glass transition temperature, the sheet precursor becomes so soft that it folds under its own Fig. 2 The robotic arm holds a workpiece and can change its position and orientation. The laser head can move horizontally. The four low-cost cameras (highlighted by rectangles) of the vision based surface deformation measurement system are distributed around the workpiece weight. The robotic arm can also enable creating more complex kirigami shapes by changing the orientation of the workpiece (see Section 5 for more details).
To achieve a desired kirigami shape, we design sequential steps of laser kirigami cutting and folding processes (one illustrative example shown in Fig. 4), which is treated as inputs into the control software. The software then controls the motions of the robotic arm and the laser head. For each folding process, there is a closed-loop controller which controls the laser head speed and the laser beam power (see the details in Section 4), with the feedback provided by the vision-based monitoring system, which can keep measuring the surface deformation of the sheets in real time.

Vision-based monitoring system for tracking surface morphology evolution
As one core subsystem in the laser kirigami platform, the real-time vision-based monitoring module (see Fig. 5) consists three major components, a multi-camera setup (with installed webcams surrounding the sheet precursors), a developed algorithm for in-process calculation of the surface morphology evolution [32,33], and the detachable planar markers for labeling the facets of the kirigami sheet precursors. Note that there are several requirements for the planar markers associated with the sheet precursor origami design: (1) each facet of the 3D kirigami object should be Fig. 3 (a) The distance between the laser lens and a workpiece is greater than the laser lens' focal length, the laser beam is defocused and heats the workpiece along a designed folding path; when the temperature of the workpiece along the folding path surpasses the glass transition temperature, the sheet precursor becomes so soft that it folds under gravity. (b) When the distance is equal to the laser lens' focal length, the laser beam is well-focused on the region of incidence on the workpiece and thus cuts the workpiece. In our system, we use a 4axis robotic arm holding the workpiece in order to adjust the distance between the laser lens and the workpiece Fig. 4 The sequential folding processes for a kirigami box could be achieved through the followings operations: (a) clamp the previously cut 2D sheet precursor with markers on the robotic arm; use defocused laser beam to heat the workpiece along the folding paths 1, 2, and 3 (see Fig. 11) to bend the sheet into the desired shape in (b), (c), and (d) respectively; slightly rotate the workpiece by 45 • towards the laser nozzle, and heat the folding path 4 to create the shape in (f); heat the folding path 5 of the shape in (f) and then cut the workpiece along the cutting path 6 to form a closed box (g) attached with sufficient fiducial markers (2 to 4 markers based on investigations reported in [32]); (2) the non-curved crease patterns are applied for the current kirigami design, so the boundaries of each facet are piecewise linear, and consequently, such facets are placed on a 2D-plane under the folding rules for each crease. Therefore, the planar markers can sufficiently represent the plane lines of each segment of the kirigami workpiece.
The multi-camera setup includes four low-cost digital cameras (3 megapixels with fixed focal length). The cameras are positioned around the sheet precursors (see Fig. 2). The schematic diagram in Fig. 5 describes the procedures for the vision-based monitoring approach. First, the process table was generated to list a sequence of creases for sheeting folding process(es). For every step in the process table, the vision approach prefers a collection of installed vision sensors (webcams) to monitor the process. As such, the vision system has a broad field of view to capture all facets and creases, and measure the angles between facets to realize designated geometries in upcoming sheet folding process(es). Our earlier investigations suggest that the vision based approach allows a frame rate at 5 ∼ 10 fps (frames per second) for tracking fiducial markers [34]. It then can track the morphology evolution with a maximal bending rate around 10 degree/sec [32,33]. With the multi-camera setup, the vision-based tracking approach allows an error rate within 5% in measuring dihedral angles in folding processes. Hence, the vision based real-time measuring system provides an accurate quantifier (in terms of geometric tolerance) as the feedback to the closed-loop control system.

The closed-loop controller
The diagram in Fig. 6 illustrates the closed-loop control for the laser bending process. Take a simple bending (shown in Fig. 7) as an example. To obtain a desired bending angle θ d between facet A and B, the supervisor of the control system sets the bound for the desired bending angle as θ d , and the vision system measures the bending angle θ k−1 at time step k − 1. The bending angle error is e k−1 = θ d −θ k−1 . We develop two independent controllers to obtain the desired bending angle: (1) a PID controller determines the laser intensity for the next sampling period based on the proportional, integral and derivative bending angle error; (2) a derivative controller (D controller) determines the laser scanning speed at the next sampling time based on the proportional and derivative bending angle errors. Both the laser intensity and the scanning speed can affect the angular velocity of the bending angleθ k . Ifθ k is very small, the folding process will take a long time; ifθ k is very large, the folding process is hard to control and the accuracy of the folding angle may not be guaranteed. Therefore, the laser intensity and scanning speed should be carefully set to achieve the desired folding accurately.
There are two computations needed for the PID controller to adjust the laser intensity. The first step is to calculate the derivative of the laser intensity at time k, ΔI k , based on the proportional, accumulated and derivative error values of the bending angle, where k P I ntensity , k I I ntensity , and k D I ntensity are the proportional, accumulated, and derivative parameters of the PID controller respectively, and their values are 0.2, 10, and 0.004 respectively; e k−1 is the error of the bending angle at the previous time step; k−1 i=0 e i Δ t i is the summation of all the past errors weighted by their time duration;ė k−1 is the derivative of the bending angle, and I k−1 is the previous laser intensity.
The second step is to determine the value of the laser intensity I k at step k based on the pre-calculated derivative of the laser intensity ΔI k as follows, where I step is the laser intensity step-size which is equal to 10% of the full laser power, and I k is the next laser intensity. We limited the maximum value of the laser intensity used for bending to 40% of the full laser power to prevent the workpiece from burning. The controller gradually increases the laser power up to the maximum power to avoid undesired deformation around the heating line caused by radical temperature changes. During the folding process, the controller controls the laser intensity so that the angular velocity of the bending angle is neither too fast nor too slow. As the bending angle approaches the target angle, the controller uses lower laser power to reduce the falling speed not only for attaining an accurate bending angle, but also for preventing undesired overshoot that is irreversible. The D controller adjusts the laser scanning speed according to the error rate of the bending angle. First, the derivative of the laser scanning speed is determined as, where k D speed is the derivative parameter of the derivative controller, which is 2000; S base is the minimum value of the laser scanning speed, which is 3000 mm/min, and S k−1 is the laser scanning speed at the previous time step. Second, the value of the next laser scanning speed is increased or decreased by the step-size of the speed variation according to whether the pre-calculated derivative of the laser scanning speed is bigger than or equal to the step size, or smaller than zero, respectively. Otherwise, it remains the same. The step-size of the speed variation is Fig. 6 The core logic controlling the defocused laser beam to bend workpieces, where θ k−1 is the observed bending angle, θ d the desired bending angle, P k the laser power, and v k the laser scanning speed Supervisor Controller D Actuator Plant Vision track system Fig. 7 The sheet forming process: the top figure shows the planar sheet with markers pasted on its surface at the initial step. The defocused laser beam scans and heats the workpiece following the designed folding path represented by the dashed line; then the part around the dashed line becomes so soft that the gravity of facet B can bend the sheet S step = 1000 mm/min. The scanning speed S k is then determined as In each control interval, the controller adjusts the laser scanning speed to enable it to compensate for the error rate of the bending angle in a prompt manner. This is done by reducing the length of the control interval, as appropriate, during the bending process.

Creating kirigami shapes by employing the robotic arm
We employ a Dobot Magician robotic arm with 4 degrees of freedom as the sheet precursor fixture placed in the laser machine coordinates. The position repeatability is 0.2 mm. The robotic arm can change the distance between the workpiece and laser lens, and the orientation of the workpiece (see Fig. 3). In conventional laser systems, a workpiece usually stays horizontally, and the maximum bending angle around the lateral axis (dashed line) is 90 • when the gravity is the only external load for folding the workpiece. In our work, when the robotic arm rotates the workpiece around the lateral axis by an angle β, the maximum folding angle around the lateral axis of the workpiece can be extended to 90 • + β (see Fig. 8).
The robotic arm can also rotate the workpiece around the perpendicular axis (the line perpendicular to the plane of the undeformed workpiece). In this way, the robotic arm expands the possible deformation patterns, and more complicated 3D shapes can be formed compared to the conventional laser systems without using robotic arms.

Laser scanning path design
One should notice that an improper laser scanning path may cause defects on a workpiece. As shown in Fig. 9 (a), there are two distinct melting points near the two ends of a crease, because, during the folding process, a defocused laser beam scans the folding path back and forth precisely following the folding path. Also, during the transition in the laser head motion, the laser nozzle first slows down in the current direction, then temporarily Fig. 8 The workpiece is horizontally clamped at an initial step; the dashed line represents the folding path; the gravity G can fold the workpiece when the laser heats the folding path. If the workpiece is kept in a horizontal plane, the maximum bending angle is 90 • (left); if the robotic arm rotates the workpiece by β degrees, the maximum bending angle can be extended to β + 90 • (right) Fig. 9 (a) The hot spots cause the melting areas around two ends of the crease when the laser scanning path precisely follows the folding path. (b) A good sample without noticeable melting points results when the laser scanning path follows the scheme shown in Fig. 10 stops moving, and then moves to the opposite direction, so the laser beam keeps heating the area around the end of a crease during the process of changing the laser head motion's direction. Therefore, the temperature around the two ends is further increased and rises higher than the temperature along the rest of the folding paths, which creates two melting points near the two ends of the crease.
To avoid these hot spots, we design a new folding path (see Fig. 10) for each pure bending. In this way, we can effectively avoid the melting points since the duration for which the laser beam heats the area around the ends of creases is reduced. Figure 9 shows the result as evidence that the applied optimal laser scanning path pattern can significantly reduce the overheating effects on the crease edges. The schematic diagram for folding a box is shown in Fig. 11. A planar sheet is cut into the shape plotted by the solid lines in Fig. 11(a); the dashed lines represent the folding path; the numbers indicate the folding order. Figure 11(b) shows the top and front views of the workpiece during the folding step 4; we use the robotic arm to rotate the workpiece by an angle β so that the gravity, represented by G, can fold the workpiece into the desired shape. The procedure for heating crease 5 is similar to step 4. The final step is to cut the  The laser-cut sheet precursor is clamped on the robotic arm. The dashed line segments 1∼5 represent the folding paths (creases to be formed) to form the box. The dash-dot line segment 6 represents the cutting path after the box is formed dash-dot line on the workpiece. Figure 12 shows the previously cut workpiece that is clamped on the robotic arm, and Fig. 4 shows the folding steps to form a closed box. The finished kirigami box is shown in Fig. 13.

Conclusion
In this work, we present a laser-based autonomous lowvolume custom manufacturing system that can fabricate 3D objects by sequentially cutting and folding 2D plastic sheets in full automation. The robotic arm allows the adjustment of the distance between the workpiece and laser lens so that the laser source offers two different functionalities, viz., sheet cutting and folding. In addition, the robotic arm with 4 degrees of freedom can adjust the orientation of the workpiece to extend the bending shapes. To develop the closed-loop control system, the low-cost vision-based surface deformation measurement technique composed of four low-cost cameras is introduced as a non-intrusive monitoring approach to track the folding process in realtime. The experiment on folding a closed box shows the performance of the laser kirigami system in full automation.
In the future, we will introduce 6-axis robotic arms for achieving complicated folding (e.g., bending angle θ = 180 • ), and improve the precision of the current laser origami platform's accuracy for realizing complicated geometries of the kirigami products. Also, we will develop software sos that the laser kirigami can automatically convert the desired kirigami shape to manufacturing processes.