Multi-Task Mixture Density Graph Neural Networks for Predicting Cu-based Single-Atom Alloy Catalysts for CO2 Reduction

doi:10.21203/rs.3.rs-2186235/v2

Download PDF

Article

Multi-Task Mixture Density Graph Neural Networks for Predicting Cu-based Single-Atom Alloy Catalysts for CO₂ Reduction

https://doi.org/10.21203/rs.3.rs-2186235/v2

This work is licensed under a CC BY 4.0 License

Version 2

posted

You are reading this latest preprint version

Graph neural networks (GNNs) have drawn more and more attention from material scientists and demonstrated a strong capacity to establish connections between the structure and properties. However, with only unrelaxed structures provided as input, few GNN models can predict the thermodynamic properties of relaxed configurations with an acceptable level of error. In this work, we develop a multi-task (MT) architecture based on DimeNet++ and mixture density networks to improve the performance of such task. Taking CO adsorption on Cu-based single-atom alloy catalysts as an example, our method can reliably predict CO adsorption energy with a mean absolute error of 0.087 eV from the initial CO adsorption structures without costly first-principles calculations. Further, compared to other state-of-the-art GNN methods, our model exhibits improved generalization ability when predicting catalytic performance of out-of-domain configurations, built with either unseen substrate surfaces or doping species. The proposed MT GNN strategy can facilitate the catalyst discovery and optimization process.

Physical sciences/Materials science/Theory and computation/Atomistic models

Physical sciences/Materials science/Materials for energy and catalysis/Electrocatalysis

Machine learning

CO2 reduction reaction

Graph neural network

Multi-task learning

Mixture density network

CO₂ reduction reaction (CO₂RR), which plays an important role in alleviating environmental problems caused by the greenhouse effect ¹, has aroused a widespread research interest. Numerous attempts have been made to comprehend the underlying mechanism of CO₂RR ^2–5, and it has been found that the adsorption energy of CO (${E}_{CO}$) acts as a good descriptor for the CO₂RR performance of catalysts ^{6, 7}. The vast choice of different catalysts renders the exploration of full configuration space of CO adsorption impractical. Fortunately, machine learning (ML) ⁸ has demonstrated a promising potential for accelerating the process of the design and optimization of catalysts, based on high-throughput ab-initio calculations ^9–13. Until now, many ML models have been proposed to predict ${E}_{CO}$ in order to guide the design of high-performance CO₂RR catalysts, including ensemble learning algorithms based on decision trees ^14–18 and artificial neural networks ^{19, 20}.

In contrast to traditional feature-based algorithms, graph neural networks (GNNs) ^21–24 have shown better performance and interpretability ^25–27, due to the analogy between the constructed data structure and real geometric configurations ²⁸ and the introduction of message-passing processes ²⁹. The early application of GNNs to the prediction of chemical properties ³⁰ demonstrates superior performance in cheminformatics compared to conventional methods. Following this, various GNN models, such as DTNN ³¹, SchNet ³², HIP-NN ³³, CGCNN ³⁴, MEGNet ³⁵, PhysNet ³⁶, AttentiveFP ³⁷, DimeNet ^{38, 39}, PAINN ⁴⁰, GemNet ⁴¹, HermNet ⁴², etc., are proposed with the prediction accuracy continuously improving. However, few models can predict the target adsorption energies for relaxed structures directly from initial unrelaxed structures with an acceptable accuracy ^43–45. Besides, conventional models show poor performance when extrapolating to untrained configurations, severely limiting their applicability.

In this work, we develop a new GNN model based on DimeNet + + ³⁹ to investigate efficient single-atom alloy (SAA) catalysts, which are among the most promising catalysts for CO₂RR ^46–53, significantly accelerating the catalyst optimization procedure by circumventing first-principles calculations. We aim to use initial unrelaxed structures as input to accurately predict ${E}_{CO}$ on the corresponding relaxed surfaces by a multi-task (MT) architecture ^{54, 55} with two branches for two distinct but interconnected tasks. Mixture density networks (MDNs) ^{56, 57} are further integrated into one of the branches to incorporate relaxed structure information in the training process, while the information is no longer needed in the inference process. The proposed model outperforms representative benchmark GNNs and conventional algorithms on a dataset containing 3075 CO adsorption energies on different Cu-based SAAs obtained by density functional theory (DFT) calculations. Meanwhile, a good extrapolation performance is shown on catalyst configurations built with either unseen substrate surfaces or doping species. In addition, the significance of MDNs is examined in our experiments.

Multi-task mixture density DimeNet++

Directional Message Passing Neural Network ³⁸, or DimeNet for short, is one of the state-of-the-art GNN methods for predicting the properties of molecules and materials. By default, DimeNet is comprised of one Embedding Block (EB), constructing initial directional edge features (or embeddings) based on the node features (or embeddings), and six Interaction Blocks (IBs), performing updating and aggregation of edge embeddings. DimeNet extends the concept of message-passing from node embeddings to directional edge embeddings, therefore enhancing the ability to differentiate among various molecules. As an improved version of DimeNet, DimeNet + + ³⁹ drastically decreases the training time needed while maintaining high accuracy by replacing the bilinear layer with element-wise multiplication and adding more multilayer perceptrons (MLPs). Additionally, the number of IBs used in the algorithm has also been reduced from six to four.

Our proposed model’s architecture, named multi-task mixture density (MT-MD) DimeNet++, is illustrated on the right panel of Fig. 1a, with DimeNet + + shown on the left for comparison. The model is designed under the MT learning framework, comprised of two branches for two distinct but related objectives. The main branch (MB) is used to predict adsorption energy, while the structure relaxation branch (SRB) is used to predict the change of atomic positions after relaxation in the form of bond length change. It is reasonable to assume that the change of pair-wise distance among atoms follows a distribution rather than a set of fixed values. And MDNs ⁵⁶, a type of specially designed artificial neural networks for approximating conditional distribution of labels, are applied in SRB. Additionally, MB and SRB are separated into two branches at the beginning of the architecture, and cross-stitch units (CSUs) ⁵⁸, a state-of-the-art MT learning technique, are applied to build a connection between the two branches, in order to determine the ideal network architecture for each task to prevent possible negative transfer.

Benchmarks on basic adsorption energy prediction tasks

A dataset containing 3075 Cu-based SAAs and corresponding ${E}_{CO}$ ¹⁷ is applied in this work, in which 41 different element species are selected as dopants. By varying the surface index, the doping position, and the adsorption site of the CO molecule, 75 distinct structures are constructed for each doping species, composed of 6, 8, 8, 27, and 26 structures on the Cu(100), Cu(110), Cu(111), Cu(210), and Cu(411) surfaces, respectively. The whole dataset is split 10 times randomly with 60% being the training set, 20% being the validation set, and the remaining 20% being the test set, respectively. Every model in the tests in this section is trained and evaluated on the same 10 splits of the dataset to make a fair comparison.

Two fundamental tasks are depicted in Figs. 1b and 1c, employing initial and relaxed structures as inputs to GNNs to predict ${E}_{CO}$ obtained after optimization, respectively, in which the initial structures are defined as the unrelaxed ones. The two tasks are comparable to the IS2RE task and S2EF task defined in Open Catalyst Project (OCP) ⁵⁹, excluding force. The former is referred as the "I2I" task because both the training and test sets contain initial structures, whereas the latter is named the "R2R" task because both the training and test sets contain relaxed structures. In addition, we establish the "R2I" task, where models are trained to predict adsorption energies using relaxed structures as inputs, but use initial structures to predict corresponding energies during inference. In all three tasks, the labels used are energies obtained after DFT structural relaxation.

The dataset is benchmarked by performing the three tasks with several representative baseline models. Table 1 presents a comprehensive comparison of the 8 models on I2I and R2R tasks, in which AEVR. and STD. denote the average and standard deviation of evaluation metrics on 10 test runs, respectively. The STDAE means the standard deviation of absolute errors of test results, and a high STDAE implies that some energies estimated by the model depart significantly from their actual values calculated by DFT, indicating a weak generalization ability. The metrics "within 0.02" and "within 0.1" represent the ratio of samples in test sets predicted with an absolute error smaller than 0.02 and 0.1 eV, respectively. Three symmetry-function (SF) ^{60, 61} based Gradient Boosting Regression (GBR) ⁶² models are listed as representatives of traditional ML algorithms, with doping and carbon atoms chosen as centers to create the SFs. Details of constructing SFs and the hyper-parameters of GNNs are available in Supplementary Note 1 and 2. The results show that all models have a higher accuracy when predicting the adsorption energy with relaxed structures instead of their initial counterparts, which is consistent with earlier works ^{43, 59}. GBR models have a substantially lower average MAE than SchNet and CGCNN on both I2I and R2R tasks, but their STDAE values are larger than those of GNN models, indicating that GBR models can only perform well on some of the data points. In comparison, DimeNet-based models perform the best on all metrics, though the run time and the number of trainable parameters exceed those of SchNet and CGCNN. In R2R task, DimeNet is superior to DimeNet + + with the original hyper-parameters (embedding size = 128, output embedding size = 256), but the performance of DimeNet + + with the embedding size = 256 and output embedding size = 192 (denoted as DimeNet++ (emb256)) is identical with or slightly better than that of DimeNet. Hereafter, the DimeNet++ (emb256) is applied for further experiments.

Table 1

Comparison of models on I2I, MT I2I and R2R tasks.
Task	Model	Test MAE AEVR. (eV)	Test MAE STD. (eV)	Test STDAE AEVR. (eV)	Test within 0.02 AVER.	Test within 0.1 AVER.	Run Time AVER. (s)	Number of Parameters
I2I	Localized cos ⁶¹ + GBR	0.120	0.0178	0.457	0.299	0.758	6.83	\
	Gaussian-cos ⁶⁰ + GBR	0.130	0.0192	0.482	0.271	0.738	6.82	\
	Gaussian-tanh ⁶⁰ + GBR	0.132	0.0201	0.485	0.268	0.735	6.71	\
	SchNet ³²	0.257	0.113	0.421	0.103	0.409	1845.45	541697
	CGCNN ³⁴	0.182	0.0887	0.401	0.147	0.557	1712.22	703361
	DimeNet ³⁸	0.099	0.0157	0.358	0.329	0.792	7741.72	2100070
	Dimenet + + ³⁹	0.0966	0.0146	0.350	0.312	0.802	5335.63	1887110
	DimeNet++ (emb256)	0.0938	0.0148	0.356	0.337	0.821	5906.27	3545542
MT I2I	MT-MD DimeNet++ (MB1-SRB1)	0.0926	0.0149	0.347	0.348	0.813	4421.92	2099199
	MT-MD DimeNet++ (MB3-SRB2)	0.0875	0.0135	0.310	0.384	0.823	7258.57	4569855
	MT-MD DimeNet++ (MB4-SRB3)	0.0870	0.0114	0.303	0.398	0.829	9231.51	6250431
R2R	Localized cos ⁶¹ + GBR	0.116	0.0237	0.585	0.311	0.792	26.25	\
	Gaussian-cos ⁶⁰ + GBR	0.108	0.0138	0.594	0.361	0.800	25.42	\
	Gaussian-tanh ⁶⁰ + GBR	0.117	0.0122	0.643	0.358	0.801	26.03	\
	SchNet ³²	0.183	0.150	0.386	0.201	0.628	1904.08	541697
	CGCNN ³⁴	0.139	0.102	0.355	0.198	0.683	1780.57	703361
	DimeNet ³⁸	0.0556	0.0138	0.312	0.569	0.908	7678.62	2100070
	Dimenet + + ³⁹	0.0603	0.0116	0.315	0.466	0.898	5171.22	1887110
	DimeNet++ (emb256)	0.0551	0.0137	0.312	0.544	0.911	5893.75	3545542

Supplementary Tables 1 and 2 show the test results achieved by models trained using normalized training labels for the I2I and R2R tasks, respectively. Because the normalization approach could not guarantee a consistent performance increase for all algorithms, the labels of training sets in the following experiments are not normalized during the training phase. Results of R2I task, shown in Supplementary Table 3, exhibit a large MAE for all models. This proves that training models on relaxed configurations to distinguish new catalysts by their initial structures can be an ineffective way, due to a limited sampling of the potential energy surface ⁴⁵.

Adsorption energy prediction with MT GNN models

Most ML models, including GNNs, cannot predict ${E}_{CO}$ with a high accuracy for I2I task. To solve such a challenge, an MT learning framework is proposed, whose workflow is illustrated in Fig. 1d. In these tests, initial structures are applied as the inputs to predict ${E}_{CO}$, and the information of relaxed structures is included during training by adding a new branch to predict the change of bond length. The trained models are then applied to predict directly ${E}_{CO}$ using initial structures without relaxation. Supplementary Table 4 compares the performance of DimeNet + + and MT-MD DimeNet + + with multiple architectures on the 10 test sets of I2I task, and results of two MT-MD DimeNet + + models are also listed in the "MT I2I" category in Table 1. In these tables, the numbers behind ‘MB’ and ‘SRB’ indicate the numbers of IBs used in MB and SRB, respectively.

As is shown in Supplementary Table 4, all MT-MD DimeNet + + models outperform original DimeNet++. Notably, even the model utilizing only one IB in MB and SRB (MB1-SRB1, also shown in Table 1), where both the number of trainable parameters and run time are less than those of DimeNet++, has an improved prediction accuracy with an average MAE of 0.0926 eV. These results clearly show the superiority of our method. The best model on the MT I2I task is MB4-SRB3 with an MAE of 0.0870 eV and 9231.51s needed to train for once. To balance the performance and computational costs, MB3-SRB2 with an MAE 0.0875 eV is chosen for further discussion.

Extrapolation on different surfaces

It is now well-known that different Cu facets lead to different products for CO₂RR ^63–66. More and more experiments are designed to realize a high product selectivity by synthesizing Cu nanostructures with specific facets. Accordingly, a good extrapolation capability on different surfaces is the key to applying ML models to facilitate the optimization of CO₂RR catalysts. The demonstration of the generalization capacity of our model is proceeded by training the model to predict ${E}_{CO}$ of SAAs on untrained surfaces.

In our dataset, there are 246, 326, 328, 1107, and 1066 samples based on Cu(100), Cu(110), Cu(111), Cu(210), and Cu(411) surfaces, respectively. By randomly selecting 246 samples built upon each surface, five subsets are constructed. One of the five subsets is chosen as the target, which is further split half-and-half as the validation and test sets, while the remaining 4 subsets are combined as the training set, leading to a dataset with an 8: 1: 1 split ratio. Accordingly, the number of training samples is the same for different surface extrapolation tasks. Based on these datasets, MT-MD DimeNet + + is trained to predict ${E}_{CO}$ on SAAs which are built on surfaces that do not appear in the training sets. The process is conducted 10 times with the same 10 random seeds for each of the five surfaces, and the average MAEs are depicted in Fig. 2a, with detailed results reported in Supplementary Table 5. Although trained on a relatively small dataset (984 samples in total), our model still achieves average MAEs below 0.2 eV on 4 out of 5 test sets. For the Cu(110) surface, we observe that the high MAE is caused by 4 outliers from the Sb-doped case, where the DFT calculated ${E}_{CO}$ is larger than 10 eV. These apparently unphysical values are attributed to a great reconstruction of the subsurface structure after optimization, shown in Supplementary Fig. 1, which is not the focus of our work. The data points are not removed in previous tests to avoid data leakage.

To further demonstrate the extrapolation ability of MT-MD DimeNet++, we evaluate our model by predicting ${E}_{CO}$ on SAAs based on Cu(751), a surface that has been proved to be more selective for C-C coupling than Cu(100) ⁶⁷. We choose 4 doping species (Ag, Zn, Ga and Ge), 2 from d-block and 2 from p-block, which are the most promising doping elements to build Cu-based SAA catalysts for CO₂RR due to their high activity and selectivity, high stability, non-toxicity and low cost ^{17, 68, 69}. Together with 9 initial structures for each doping case, 36 SAA configurations on Cu(751) are calculated in total to build a new test set. The 4 outliers of Sb-doped Cu(110) mentioned above are removed from the original SAA dataset, and the remaining 3071 samples are applied as the training set. The results in Fig. 2b show that both the training and test MAEs are below 0.1 eV, and most predicted values are within the acceptable 0.1 eV error threshold (71.8% for the training set and 69.4% for the test set), reflecting an extraordinary extrapolation capacity of our model.

Additionally, to compare MT-MD DimeNet + + with baseline algorithms, all models are then trained on the dataset, which has been shuffled with 5 different random seeds, to perform the Cu(751) extrapolation test for 5 times, with results shown in Supplementary Table 6. Three DimeNet-based models show the lowest MAEs compared with other models, and MT-MD DimeNet + + is the only one with an MAE below 0.09 eV (0.0893 eV).

Extrapolation on different elements

The doping species have a great impact on the chemical properties of SAAs, leading to different substrate-CO interaction behaviors. The ability to predict ${E}_{CO}$ on SAAs with unseen doping species is another important aspect to accelerate the development of new SAA catalysts. In this section, each of the 41 doping elements is applied as the target element, respectively, to construct 41 datasets. In each dataset, 75 SAAs doped by the target atom form the test set, while the remaining samples, in which the target element does not appear, form the training set. When Sb is not the target element, the training set contains 2996 data points, excluding 4 Sb-containing samples with ${E}_{CO}$ larger than 10 eV, whereas the number of training data is 3000 for the case with Sb as the target element. MT-MD DimeNet + + performs the prediction task for 5 times on each target element with 5 different random seeds, and the results are shown in Fig. 3a in a periodic table with more details provided in Supplementary Table 7. In total, the average test MAEs of 11 out of 41 elements are within the range of acceptable error (0.10–0.15 eV). The model can predict most untrained non-metal elements with a relatively high accuracy. Although the model performs the best for Cd and Pd (with an MAE of 0.100 and 0.101 eV, respectively), it does not show a good performance when extrapolating to some transition metal elements, especially those in VIB, VIIB groups and the first two columns of VIIIB group. Our preliminary interpretation for these phenomena is that the relatively higher complexity of the interaction pattern between the transition metal elements with Cu substrates benefits the extrapolation on the simpler non-metal doping elements. We leave the study of the underlying mechanism in our future work.

In order to further assess the generalization ability, a series of transfer learning tasks are designed for the three target elements with the worst performance (except Sb), i.e., N, Re and Ru. In this experiment, 2, 5, 10, 20, and 30 data points from the initial test sets are randomly selected and added to the training set to build transfer learning tasks with decreasing levels of difficulty. For each task, the sampling is repeated 5 times based on 5 different random seeds, resulting in 25 datasets in total for each element. All of the baseline models are applied to make a comparison with MT-MD DimeNet++, and the average MAEs for different tasks are plotted in Figs. 3b-d, details of which are available in Supplementary Tables 8–10. Experiments in which no extra data points are supplemented are also conducted for baseline models, and the results are also derived as an average of 5 tests for each model. None of the models can predict ${E}_{CO}$ on N-doped SAAs accurately, and the three GBR-based models even perform worse when two N samples are included. It is noted many ${E}_{CO}$ on N-doped samples (23 out of 75) are smaller than − 1.0 eV, largely deviated from the mean of the ${E}_{CO}$distribution for the whole dataset. This indicates that there is an unusual pattern of interaction between CO and N-containing catalysts suggested by the reconstructed structures after optimization, leading to the failure of extrapolation. Compared with N-doped cases, test MAEs on Re- and Ru-doped samples show a gradual decrease as the number of Re/Ru samples added increases. For Re-doped SAAs, MT-MD DimeNet + + is the only model achieving an average MAE lower than 0.1 eV (0.0881 eV) when only 10 Re samples are trained, and it continues to show the best performance till 30 extra samples are included (0.0739 eV), notably better than DimeNet++ (0.0895 eV). For Ru doped SAAs, the transfer is relatively easy compared with the other two cases. DimeNet++ (0.0533 eV) outperforms MT-MD DimeNet++ (0.0657 eV) for datasets including 10 Ru samples, while in the task with 30 Ru samples, the MT-MD one is still better than DimeNet + + with an MAE of 0.0310 eV versus 0.0356 eV. Therefore, although MT-MD DimeNet + + cannot directly predict ${E}_{CO}$ of SAAs doped with untrained species, the model reaches a high accuracy as long as around 5 to 10 extra samples are included as part of the training samples, largely reducing the computational cost.

The contribution of MDNs in MT-MD DimeNet + + on its improved performance is explored by comparing MT-MD DimeNet + + to multi-task MLP (MT-MLP) DimeNet++, in which the MDN layers at the end of MT-MD DimeNet + + are substituted with MLPs. The test results of different models for the MT I2I task described in Fig. 1d is shown in Table 2, and all of the results are again the average of 10 tests. MDNs do have an advantage over MLPs for the structure relaxation prediction task, suggested by a better performance of MT-MD model than that of MT-MLP model of the same architecture. As a result, the assumption that the bond length change is more likely to follow a distribution rather than being deterministic values is justified.

Table 2

Comparison of different types of MT DimeNet++.
Type of Model	Architecture	Test MAE AEVR.	Test MAE STD.	Test STDAE AEVR.	Test within 0.02 AVER.	Test within 0.1 AVER.	Run Time AVER.	Number of Parameters
MT-MLP DimeNet++	MB3-SRB2	0.0889	0.0150	0.351	0.378	0.828	6773.94	4569735
MT-MD DimeNet++	MB3-SRB2	0.0875	0.0135	0.310	0.384	0.823	7258.57	4569855

The function of dopant features in the networks is further investigated to understand the essential role of dopants in the catalytic performance of SAAs. First, the significance of dopant features and adsorbed atom features in GBR models is studied by only retaining one of them as the structure information of the configuration. Compared with the original model using both information to represent structures, the results in Supplementary Table 11 show that the excellent performance of SF-based models is attributed to dopant features rather than adsorbed molecule properties. Accordingly, the conventional GNN algorithms are further modified by adding the updated node embeddings of dopants directly to the resulting graph-level crystal embeddings, in order to emphasize the impact of dopants. It turns out that such a simple adjustment in the networks can lead to an improved performance, as illustrated in Supplementary Fig. 2. For DimeNet-based modes, the embeddings of incident edges of dopants are averaged to obtain the dopant node embeddings and added to all edge embeddings of corresponding crystal graphs in each IB. However, the test results do not show a notable enhancement from those of original models. A better approach to emphasize the dopants in the edge-based message-passing models remains to be explored.

In summary, we propose a novel form of GNN model named multi-task mixture density (MT-MD) DimeNet + + to predict ${E}_{CO}$ of Cu-based SAAs, optimizing SAA catalysts for CO₂RR. In the model, empowered by MDNs, an SRB is applied to predict the change of atomic positions after optimization, and the performance on ${E}_{CO}$ prediction task has improved notably when the relaxed structure information is encoded into the weight parameters during training. The model also exhibits extraordinary capacity on surface extrapolation tasks, so that ${E}_{CO}$ of SAAs constructed on high index Cu surfaces can be predicted accurately using the model only trained on low index data. Transfer learning to unseen doping species is also feasible with only 5–10 extra SAAs samples doped by the test element provided, which can save a large amount of computational costs. Further experiments have been conducted to demonstrate the contribution of MDNs on the improved performance. It can be expected the presented framework can be applied with other advanced GNN models to predict either thermodynamic adsorption energy or kinetic barrier of different reaction processes to tackle the catalyst design challenge.

Details of Density function theory (DFT) computation for building the SAA dataset and training of ML models

The DFT calculations were performed using Vienna ab initio simulation package (VASP) ^{70, 71}, with the project-augmented wave potential and a 400 eV cut-off energy. Revised Perdew–Burke–Ernzerhof (rPBE) ⁷² was used to describe exchange and correlation interaction within the generalized gradient approximation (GGA). For magnetic systems, spin-polarized computations were carried out. A (3 × 3 × 1) Monkhorst-Pack mesh for k-point sampling was performed for Brillouin-zone integration, and the smallest space between two k-points was set as 0.3 Å⁻¹. The vacuum distances were set larger than 20 Å, and the Cu atoms at the bottom two layers were frozen during relaxation, until the convergence threshold, i.e., 1 × 10^− 4 eV/atom for energy and 0.05 eV/atom for force, was reached. After that, ${E}_{CO}$ was derived from energies of relaxed systems by subtracting the energies of CO and bare SAA from the energy of SAA adsorbed with a CO molecule. The ab initio molecular dynamics (AIMD) was done at 300K with an NVT ensemble using a Nosé-Hoover thermostat ^{73, 74} for 10 ps with a 1.0 fs time step.

All of the GNN models were trained under the framework implemented by OCP ⁵⁹ to make a fair comparison, and the GBR algorithm was applied using the Scikit-learn package ⁷⁵. All of the GNN models in this work, including MT ones, were trained on a NVIDIA T4 GPU, and the GBR based models were done on an Intel i7-8700 CPU core.

Directional message passing in DimeNet and DimeNet++

When updating the directional embedding of a single edge in DimeNet, two pieces of information are incorporated: the edge encoding from the previous layer and the directional encoding aggregated from its incident edges. The latter includes angle data between the two edges, as well as edge length data from the incident edge. To be more precise, the directional message-passing process in the ${(l+1)}_{th}$ layer can be formulated as

$$\begin{array}{c}{m}_{ji}^{\left(l+1\right)}={U}_{t}\left({m}_{ji}^{\left(l\right)},{\sum }_{k\in {N}_{j}}{M}_{t}\left({m}_{kj}^{\left(l\right)},{e}_{ji},{a}_{\left(kj,ji\right)}\right)\right),\#\left(1\right)\end{array}$$

in which ${m}_{ji}$ denotes the directional edge embedding sent from atom $j$ to $i$, and ${N}_{j}$ denotes neighbor atoms of $j$ within the cut-off radius. ${U}_{t}$ and ${M}_{t}$ are the update function and message-passing function, respectively, as defined in ²⁹. ${e}_{ji}$ is the expansion of distance between $j$ and $i$ (${d}_{ji}$) using radius basis, and the angle between bond $ji$ and $kj$, as well as ${d}_{kj}$, are incorporated into ${a}_{\left(kj,ji\right)}$ using spherical harmonics and spherical Bessel functions of the first and the second kind, establishing 2D spherical Fourier-Bessel basis, which achieves equivariance with respect to actions in the SO(3) group ^{76, 77}. As the core part of the operation, the message-passing function can be expressed as

$$\begin{array}{c}Msg={\sum }_{k\in {N}_{j}}\left({W}_{a}*{a}_{\left(kj,ji\right)}\right)*{W}_{bilinear}*\left[\left({W}_{e}*{e}_{ji}\right)\odot {\sigma }_{m}\left({W}_{m}*{m}_{kj}^{l}+{b}_{m}\right)\right],\#\left(2\right)\end{array}$$

where $W$, $b$ and $\sigma$ represent weight, bias and the activation function in fully connected layers, respectively, and $\odot$ represents element-wise multiplication. After that, the derived message is to be combined with ${m}_{ji}^{\left(l\right)}$ and several Residual layers ⁷⁸ are applied to improve stability of the network and prevent gradient vanishing during training.

Noting that the bilinear layer is computationally costly, DimeNet + + replaces it with a simple Hadamard product with MLPs to accelerate the computation as

$$Msg={\sigma }_{up}\left({W}_{up}\left[{\sum }_{k\in {N}_{j}}{W}_{a2}*\left({W}_{a1}*{a}_{\left(kj,ji\right)}\right) \odot {\sigma }_{down}\left({W}_{down}\left({W}_{e2}*\left({W}_{e1}*{e}_{ji}\right)\odot {\sigma }_{m}\left({W}_{m}*{m}_{kj}^{l}+{b}_{m}\right)\right)\right)\right]\right).$$

With additional model architecture adjustments, DimeNet + + can reach an 8x quicker performance than DimeNet.

Mixture density networks (MDNs)

MDNs assume that the labels follow an unknown conditional distribution given the samples and use a set of Gaussians to approximate the distribution. The outputs of MDNs are not the label values, but the means, standard deviations and weight coefficients of Gaussians. The loss function of MDNs can be written as

$$\begin{array}{c}{\mathcal{L}}_{\mathcal{M}\mathcal{D}\mathcal{N}}=-log{\sum }_{n=1}^{N}{p}_{n}*{\mathcal{N}}_{\mathcal{n}}\left({y}_{true}|{\mu }_{n},{\sigma }_{n}\right),\#\left(4\right)\end{array}$$

in which ${{\mu }}_{n}$ and ${{\sigma }}_{n}$ represent the mean and standard deviation of the ${n}_{th}$ Gaussian distribution ${\mathcal{N}}_{\mathcal{n}}$, respectively, and ${p}_{n}$ denotes the contribution of the Gaussian to the real distribution. During training, the value of ${\mathcal{L}}_{\mathcal{M}\mathcal{D}\mathcal{N}}$ is reduced, so that the labels ${y}_{true}$ have a larger and larger probability to appear in the derived mixed distribution, which therefore characterizes the system.

Cross-stitch units (CSUs)

Assuming that ${f}_{I,1}$ and ${f}_{I,2}$ in the form

$$\begin{array}{c}{f}_{I,1}={\left(\begin{array}{ccc}{x}_{I,1}^{1}& \dots & {x}_{I,1}^{n}\end{array}\right)}^{T}\#\left(5\right)\end{array}$$

$$\begin{array}{c}{f}_{I,2}={\left(\begin{array}{ccc}{x}_{I,2}^{1}& \dots & {x}_{I,2}^{n}\end{array}\right)}^{T}\#\left(6\right)\end{array}$$

are the inputs of the CSU, i.e., the output edge embeddings of the two IBs above the CSU, the CSU should be a $2n*2n$ learnable matrix. The cross-stitch operation can be formulated as

$$\begin{array}{c}{f}_{O,1}={\left(\begin{array}{ccc}{\sum }_{i=1}^{n}{x}_{I,1}^{i}{a}_{i1}+{\sum }_{j=1}^{n}{x}_{I,2}^{j}{a}_{j+n,1}& \dots & {\sum }_{i=1}^{n}{x}_{I,1}^{i}{a}_{in}+{\sum }_{j=1}^{n}{x}_{I,2}^{j}{a}_{j+n,n}\end{array}\right)}^{T}\#\left(7\right)\end{array}$$

$$\begin{array}{c}{f}_{O,2}={\left(\begin{array}{ccc}{\sum }_{i=1}^{n}{x}_{I,1}^{i}{a}_{i,n+1}+{\sum }_{j=1}^{n}{x}_{I,2}^{j}{a}_{j+n,n+1}& \dots & {\sum }_{i=1}^{n}{x}_{I,1}^{i}{a}_{i,2n}+{\sum }_{j=1}^{n}{x}_{I,2}^{j}{a}_{j+n,2n}\end{array}\right)}^{T},\#\left(8\right)\end{array}$$

in which ${a}_{ij}$ denotes the element in the ${i}_{th}$ row and ${j}_{th}$ column in a CSU matrix, while ${f}_{O,1}$ and ${f}_{O,2}$ are outputs of the CSU, as well as the inputs of the two IBs below it. It can be seen that ${f}_{I,1}={f}_{O,1}$ if the CSU is an identity, and otherwise the CSU can model how to share the two sets of weight parameters in two connected IBs by searching for appropriate ${a}_{ij}$ values to control the contribution of each element of two input features on the output, therefore deciding the effective MT network architecture.

DATA AVAILABILITY

The SAA dataset that supports the findings of this study is available from the corresponding author upon reasonable request.

CODE AVAILABILITY

The DFT calculation codes used in this study were VASP (version 5.4), which is a commercial code (see www.vasp.at). The codes are available upon request to the corresponding author.

ACKNOWLEDGEMENTS

This work was supported by National Natural Science Foundation of China (Project No. 62006219), Hong Kong Innovation and Technology Fund Project No. ITS/170/20, ITS/241/21, the Guangdong Innovative and Entrepreneurial Research Team Program (grant No. 2017ZT07C341). C. Liang and B. Wang contributed equally.

AUTHOR CONTRIBUTIONS

C.L. constructed the dataset, implemented the MTMD DimeNet++ model and conducted all experiments in this work. B.W. contributed the idea of applying the multi-task learning framework and MDNs for the prediction task. S.H., G.C., P.-A.H. and X.Z. provided guidance about the experiments to be conducted to testify the effectiveness of the model and the interpretation of results. All authors contributed to the writing of the paper.

COMPETING INTERESTS

The authors declare that they have no known competing financial interests or personal relationships that could influence the work.

Appel, A. M., et al. Frontiers, Opportunities, and Challenges in Biochemical and Chemical Catalysis of CO2 Fixation. Chem. Rev. 113, 6621-6658 (2013).
Liu, X., et al. Understanding trends in electrochemical carbon dioxide reduction rates. Nat. Commun. 8, 15438 (2017).
Peterson, A. A., Abild-Pedersen, F., Studt, F., Rossmeisl, J. & Nørskov, J. K. How copper catalyzes the electroreduction of carbon dioxide into hydrocarbon fuels. Energy Environ. Sci. 3, 1311-1315 (2010).
Nie, X., Esopi, M. R., Janik, M. J. & Asthagiri, A. Selectivity of CO2 reduction on copper electrodes: the role of the kinetics of elementary steps. Angew. Chem. Int. Ed. 52, 2459-2462 (2013).
Chen, Y., et al. Ethylene Selectivity in Electrocatalytic CO2 Reduction on Cu Nanomaterials: A Crystal Phase-Dependent Study. J. Am. Chem. Soc. 142, 12760-12766 (2020).
Tran, K. & Ulissi, Z. W. Active learning across intermetallics to guide discovery of electrocatalysts for CO2 reduction and H2 evolution. Nat. Catal. 1, 696-703 (2018).
Peterson, A. A. & Nørskov, J. K. Activity Descriptors for CO2 Electroreduction to Methane on Transition-Metal Catalysts. J. Phys. Chem. Lett. 3, 251-258 (2012).
Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O. & Walsh, A. Machine learning for molecular and materials science. Nature 559, 547-555 (2018).
Mao, X., et al. Computational high-throughput screening of alloy nanoclusters for electrocatalytic hydrogen evolution. Npj Comput. Mater. 7, 46 (2021).
Pei, W., Wang, P., Zhou, S. & Zhao, J. Inverse Design of Nanoclusters for Light-Controlled CO2–HCOOH Interconversion. J. Phys. Chem. Lett. 13, 2523-2532 (2022).
Zhong, M., et al. Accelerated discovery of CO2 electrocatalysts using active machine learning. Nature 581, 178-183 (2020).
Lin, S., Xu, H., Wang, Y., Zeng, X. C. & Chen, Z. Directly predicting limiting potentials from easily obtainable physical properties of graphene-supported single-atom electrocatalysts by machine learning. J. Mater. Chem. A 8, 5663-5670 (2020).
Xu, H., Cheng, D., Cao, D. & Zeng, X. C. A universal principle for a rational design of single-atom electrocatalysts. Nat. Catal. 1, 339-348 (2018).
Panapitiya, G., et al. Machine-Learning Prediction of CO Adsorption in Thiolated, Ag-Alloyed Au Nanoclusters. J. Am. Chem. Soc. 140, 17508-17514 (2018).
Yang, Z., Gao, W. & Jiang, Q. A machine learning scheme for the catalytic activity of alloys with intrinsic descriptors. J. Mater. Chem. A 8, 17507-17515 (2020).
Chen, A., Zhang, X., Chen, L., Yao, S. & Zhou, Z. A Machine Learning Model on Simple Features for CO2 Reduction Electrocatalysts. J. Phys. Chem. C 124, 22471-22478 (2020).
Wang, D., et al. Accelerated prediction of Cu-based single-atom alloy catalysts for CO2 reduction by machine learning. Green Energy Environ., (in press, 2021).
Li, X., et al. A transferable machine-learning scheme from pure metals to alloys for predicting adsorption energies. J. Mater. Chem. A 10, 872-880 (2022).
Ma, X., Li, Z., Achenie, L. E. & Xin, H. Machine-Learning-Augmented Chemisorption Model for CO2 Electroreduction Catalyst Screening. J. Phys. Chem. Lett. 6, 3528-3533 (2015).
Chen, Y., Huang, Y., Cheng, T. & Goddard, W. A., III. Identifying Active Sites for CO2 Reduction on Dealloyed Gold Surfaces by Combining Machine Learning with Multiscale Simulations. J. Am. Chem. Soc. 141, 11651-11657 (2019).
Gori, M., Monfardini, G. & Scarselli, F. A new model for learning in graph domains. In: Proceedings. 2005 IEEE international joint conference on neural networks, 729-734 (2005).
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M. & Monfardini, G. The graph neural network model. IEEE Trans. Neural Netw. 20, 61-80 (2009).
Defferrard, M., Bresson, X. & Vandergheynst, P. Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems, (2016).
Kipf, T. N. & Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. In: International Conference on Learning Representations, (2017).
Schmidt, J., Marques, M. R. G., Botti, S. & Marques, M. A. L. Recent advances and applications of machine learning in solid-state materials science. Npj Comput. Mater. 5, 83 (2019).
Omidvar, N., et al. Interpretable Machine Learning of Chemical Bonding at Solid Surfaces. J. Phys. Chem. Lett. 12, 11476-11487 (2021).
Lu, S., et al. Coupling a Crystal Graph Multilayer Descriptor to Active Learning for Rapid Discovery of 2D Ferromagnetic Semiconductors/Half-Metals/Metals. Adv. Mater. 32, 2002658 (2020).
Trinajstic, N. Chemical graph theory, 2nd edn (CRC press, Boca Raton, 1992).
Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural Message Passing for Quantum Chemistry. In: Proceedings of the 34th International Conference on Machine Learning, 1263-1272 (2017).
Duvenaud, D. K., et al. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In: Advances in Neural Information Processing Systems, (2015).
Schütt, K. T., Arbabzadah, F., Chmiela, S., Muller, K. R. & Tkatchenko, A. Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 13890 (2017).
Schütt, K. T., Sauceda, H. E., Kindermans, P. J., Tkatchenko, A. & Muller, K. R. SchNet - A deep learning architecture for molecules and materials. J. Chem. Phys. 148, 241722 (2018).
Lubbers, N., Smith, J. S. & Barros, K. Hierarchical modeling of molecular energies using a deep neural network. J. Chem. Phys. 148, 241715 (2018).
Xie, T. & Grossman, J. C. Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. Phys. Rev. Lett. 120, 145301 (2018).
Chen, C., Ye, W., Zuo, Y., Zheng, C. & Ong, S. P. Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals. Chem. Mater. 31, 3564-3572 (2019).
Unke, O. T. & Meuwly, M. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. J. Chem. Theory Comput. 15, 3678-3693 (2019).
Xiong, Z., et al. Pushing the Boundaries of Molecular Representation for Drug Discovery with the Graph Attention Mechanism. J. Med. Chem. 63, 8749-8760 (2020).
Klicpera, J., Groß, J. & Günnemann, S. Directional Message Passing for Molecular Graphs. In: International Conference on Learning Representations, (2020).
Klicpera, J., Giri, S., Margraf, J. T. & Günnemann, S. Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules. Preprint at https://arxiv.org/abs/2011.14115 (2020).
Schütt, K., Unke, O. & Gastegger, M. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In: Proceedings of the 38th International Conference on Machine Learning, 9377-9388 (2021).
Klicpera, J., Becker, F. & Günnemann, S. GemNet: Universal Directional Graph Neural Networks for Molecules. In: Advances in Neural Information Processing Systems, 6790-6802 (2021).
Wang, Z., et al. Heterogeneous relational message passing networks for molecular dynamics simulations. Npj Comput. Mater. 8, 1-9 (2022).
Back, S., et al. Convolutional Neural Network of Atomic Surface Structures To Predict Binding Energies for High-Throughput Screening of Catalysts. J. Phys. Chem. Lett. 10, 4401-4408 (2019).
Ghanekar, P. G., Deshpande, S. & Greeley, J. Adsorbate chemical environment-based machine learning framework for heterogeneous catalysis. Nat. Commun. 13, 5788 (2022).
Gibson, J., Hire, A. & Hennig, R. G. Data-augmentation for graph neural network learning of the relaxed energies of unrelaxed structures. Npj Comput. Mater. 8, 211 (2022).
Cheng, M.-J., Clark, E. L., Pham, H. H., Bell, A. T. & Head-Gordon, M. Quantum Mechanical Screening of Single-Atom Bimetallic Alloys for the Selective Reduction of CO2 to C1 Hydrocarbons. ACS Catal. 6, 7769-7777 (2016).
Darby, M. T., Réocreux, R., Sykes, E. C. H., Michaelides, A. & Stamatakis, M. Elucidating the Stability and Reactivity of Surface Intermediates on Single-Atom Alloy Catalysts. ACS Catal. 8, 5038-5050 (2018).
Gong, L., et al. Catalytic Mechanisms and Design Principles for Single‐Atom Catalysts in Highly Efficient CO2 Conversion. Adv. Energy Mater. 9, 1902625 (2019).
Zhao, Z. & Lu, G. Cu-Based Single-Atom Catalysts Boost Electroreduction of CO2 to CH3OH: First-Principles Predictions. J. Phys. Chem. C 123, 4380-4387 (2019).
Karmodak, N., Vijay, S., Kastlunger, G. & Chan, K. Computational Screening of Single and Di-Atom Catalysts for Electrochemical CO2 Reduction. ACS Catal. 12, 4818-4824 (2022).
Yuan, Q., et al. Reaction mechanism on Ni-C2-NS single-atom catalysis for the efficient CO2 reduction reaction. J. Exp. Nanosci. 16, 255-264 (2021).
Ju, L., et al. Controllable CO2 electrocatalytic reduction via ferroelectric switching on single atom anchored In2Se3 monolayer. Nat. Commun. 12, 5128 (2021).
Yuan, H., Li, Z., Zeng, X. C. & Yang, J. Descriptor-Based Design Principle for Two-Dimensional Single-Atom Catalysts: Carbon Dioxide Electroreduction. J. Phys. Chem. Lett. 11, 3481-3487 (2020).
Caruana, R. Multitask learning. Mach. Learn. 28, 41-75 (1997).
Wang, S. H., Pillai, H. S., Wang, S., Achenie, L. E. K. & Xin, H. Infusing theory into deep learning for interpretable reactivity prediction. Nat. Commun. 12, 5288 (2021).
Bishop, C. M. Mixture density networks. Technical Report NCRG/94/004, Neural Computing Research Group, Aston University (1994).
Méndez-Lucio, O., Ahmad, M., del Rio-Chanona, E. A. & Wegner, J. K. A geometric deep learning approach to predict binding conformations of bioactive molecules. Nat. Mach. Intell. 3, 1033-1039 (2021).
Misra, I., Shrivastava, A., Gupta, A. & Hebert, M. Cross-stitch networks for multi-task learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 3994-4003 (2016).
Chanussot, L., et al. Open Catalyst 2020 (OC20) Dataset and Community Challenges. ACS Catal. 11, 6059-6072 (2021).
Behler, J. & Parrinello, M. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett. 98, 146401 (2007).
Huang, Y., Chen, Y., Cheng, T., Wang, L.-W. & Goddard, W. A., III. Identification of the Selective Sites for Electrochemical Reduction of CO to C2+ Products on Copper Nanoparticles by Combining Reactive Force Fields, Density Functional Theory, and Machine Learning. ACS Energy Lett. 3, 2983-2988 (2018).
Friedman, J. H. Greedy Function Approximation: A Gradient Boosting Machine. Ann. Stat. 29, 1189-1232 (2001).
Bagger, A., Ju, W., Varela, A. S., Strasser, P. & Rossmeisl, J. Electrochemical CO2 Reduction: Classifying Cu Facets. ACS Catal. 9, 7894-7899 (2019).
Montoya, J. H., Shi, C., Chan, K. & Nørskov, J. K. Theoretical Insights into a CO Dimerization Mechanism in CO2 Electroreduction. J. Phys. Chem. Lett. 6, 2032-2037 (2015).
Schouten, K. J. P., Qin, Z., Pérez Gallent, E. & Koper, M. T. M. Two Pathways for the Formation of Ethylene in CO Reduction on Single-Crystal Copper Electrodes. J. Am. Chem. Soc. 134, 9864-9867 (2012).
Cheng, D., et al. The nature of active sites for carbon dioxide electroreduction over oxide-derived copper catalysts. Nat. Commun. 12, 395 (2021).
Hahn, C., et al. Engineering Cu surfaces for the electrocatalytic conversion of CO2: Controlling selectivity toward oxygenates and hydrocarbons. Proc. Natl. Acad. Sci. U.S.A. 114, 5918-5923 (2017).
Jeon, H. S., et al. Operando Insight into the Correlation between the Structure and Composition of CuZn Nanoparticles and Their Selectivity for the Electrochemical CO2 Reduction. J. Am. Chem. Soc. 141, 19879-19887 (2019).
Wang, L., et al. Selective reduction of CO to acetaldehyde with CuAg electrocatalysts. Proc. Natl. Acad. Sci. U.S.A. 117, 12572-12575 (2020).
Paszke, A., et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Advances in Neural Information Processing Systems, (2019).
Fey, M. & Lenssen, J. E. Fast Graph Representation Learning with PyTorch Geometric. Preprint at http://arxiv.org/abs/1903.02428 (2019).
Hjorth Larsen, A., et al. The atomic simulation environment—a Python library for working with atoms. J. Condens. Matter Phys. 29, 273002 (2017).
Ong, S. P., et al. Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis. Comput. Mater. Sci. 68, 314-319 (2013).
Pedregosa, F., et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825-2830 (2011).
Kresse, G. & Furthmller, J. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Comput. Mater. Sci. 6, 15-50 (1996).
Kresse, G. & Joubert, D. From ultrasoft pseudopotentials to the projector augmented-wave method. Phys. Rev. B 59, 1758-1775 (1999).
Hammer, B., Hansen, L. B. & Nørskov, J. K. Improved adsorption energetics within density-functional theory using revised Perdew-Burke-Ernzerhof functionals. Phys. Rev. B 59, 7413-7421 (1999).
Nosé, S. A unified formulation of the constant temperature molecular dynamics methods. J. Chem. Phys. 81, 511-519 (1984).
Hoover, W. G. Canonical dynamics: Equilibrium phase-space distributions. Phys. Rev. A 31, 1695-1697 (1985).
Thomas, N., et al. Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds. Preprint at http://arxiv.org/abs/1802.08219 (2018).
Anderson, B., Hy, T. S. & Kondor, R. Cormorant: Covariant Molecular Neural Networks. In: Advances in Neural Information Processing Systems, (2019).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778 (2016).

(Not answered)

SupplementaryInformation.pdf

Download PDF

Version 2

posted

You are reading this latest preprint version

Multi-Task Mixture Density Graph Neural Networks for Predicting Cu-based Single-Atom Alloy Catalysts for CO₂ Reduction

Status:

Version 2

Abstract

Figures

Introduction

Results

Multi-task mixture density DimeNet++

Benchmarks on basic adsorption energy prediction tasks

Adsorption energy prediction with MT GNN models

Extrapolation on different surfaces

Extrapolation on different elements

Discussion

Methods

Directional message passing in DimeNet and DimeNet++

Mixture density networks (MDNs)

Cross-stitch units (CSUs)

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 2