More on machine learning and materials

In the last post, I looked at a paper in which machine learning had been used to predict properties of doped graphene. One of my thoughts on this was that the study seemed unsatisfying because it only concluded that the neural network could be trained to make the prediction, but there was no attempt to figure out how it made the prediction - even though that might have told us something interesting both about the network and about doped graphene.

Oddly enough the paper contained references to an interesting study where researchers have done exactly that, albeit for a very different problem. This paper by Ziletti et al, published in Nature Communications in 2018, considers the problem of finding a method to classify crystal structures that is robust and not dependent on a myriad of hand-tuned thresholds and parameters. Along the way they adapt a method for probing the internal workings of the neural network to their own application.

Admittedly, crystal structure classification doesn't sound like the most exciting problem in the world, but within materials science and condensed matter physics it is very important. A lot of materials are crystalline, i.e. they consist of periodically repeating arrangements of atoms. Knowing what these periodically repeating arrangements look like and in what ways they are symmetric is important for understanding, investigating and modelling the material - and often for figuring out what it can be useful for or how it can be improved. It is also a rather tedious process with a lot of potential for error due to noisy measurements and the fact that real-world materials are not perfect crystals, but will always contain defects of various kinds. In the study, the aim is to develop a robust classification method that can handle the presence of defects without misclassifying the structures.

The first step is to decide what sort of input data to use. This is more complex than it seems, since just using atomic positions might make the classifier inherently sensitive to defects. Instead, the researchers have chosen to use simulated diffraction patterns, which condense the information about atom placements and inter-atomic distances to a number of bright spots. (If you recall being shown diffraction in some high-school physics class, this is the same thing only with periodic atomic structures instead of the slits and using electromagnetic radiation with much shorter wavelenght). The diffraction patterns are fed into a convolutional neural networks with multiple layers, which extracts features from the patterns and then classifies the patterns based on these features. Tests of the networks show good performance, even when the data was noisy or the structures contained a high number of defects.

Now for the interesting part. As described in the previous post, feature extraction in a convolutional neural network can be likened to a process where small sections of an image are compared to a smaller image, and a positive response is given if they match. The output of the first comparison is then used in another comparison that extracts more complicated features, and so on. Training of the neural network amounts to adjusting the smaller images, or filters, to respond to features of the image that enable the network to make the correct classification. If picking out straight lines enable correct classification, at least some of the filters will end up responding to straight lines. If curves are important, some of the filters will respond to curves.

This also means that when the neural network has been trained and an image is fed into it, at some deep level in the neural network there will be a vector representing the features that are present in the image and that the neural network has been trained to extract and classify. This vector could tell us exactly what information the network is using when classifying a particular image, but due to the complexity of the preceding layers of the network it is hard to interpret. It is, however, possible to start from this representation of the extracted features and essentially go through all the layers of the network in reverse, finally arriving at a generated picture that shows just the features picked out by the network in a way that can easily be recognized by humans (these pictures are also known as attentive response maps). Using this method, the researchers found that the neural network had in fact learned to use many of the characteristics that humans use when classifying crystal structures, such as distances between atomic planes.

So why is this interesting? For one thing, it demonstrates a method of checking if the classification performed by the network is based on something we would consider significant, or if it has learned to classify based on something obviously irrelevant - say, some kind of noise that is more common in some types of images than others. It also suggests that we could use neural networks not just to make predictions or classify data points, but also to understand the differences between the data points better. It is after all entirely possible that the networks could extract some feature that we do not realize the importance of yet. Personally, I think this is the way to use machine learning in physics - not just looking for the how, but also the why.

Finally, I should mention that the method in itself is adapted from a 2018 paper on classifying X-ray images of body parts, which in turn references a much earlier paper on understanding how convolutional networks classify more ordinary images. It is perhaps telling that it was picked up in the medical field, since knowing that neural networks classify based on the right information could be vital there.

Machine learning and materials science

This post is a translation of a post that appeared on my Swedish blog in May 2019.

The other week I read an optimistic blog post on the subject of machine learning by the American skeptic and neurologist Steven Novella. He wrote, among other things, about an American research group that has trained a neural network to determine properties of doped graphene, that is graphene where some of the carbon atoms are replaced with other elements, from the placement of the dopant atoms. Novella chose to portray this as the neural network being able to perform decades of research in the course of a few days, and hinted that this could give us practical applications of graphene considerably earlier than if no machine learning had been used.

As someone who is interested in both graphene and machine learning, I obviously had to find the scientific paper the group had published and try to figure out what they had actually done.

The research question
The paper in question is published in npj Computational Materials (it is also open access, by the way) and according to the title it deals with the prediction of the so-called band gap of materials that are a combination of graphene and boron nitride. Boron nitride is a material that consists of two types of atoms, boron and nitrogen, arranged in a hexagonal lattice just like the carbon atoms in graphene. Also just like graphene, boron nitride can be produced as just a single, super-thin layer of atoms. These similarities between the two materials are a part of the reason why people try to combine them.

Another part of the reason is that while graphene has excellent electrical conductivity, it is very difficult to get boron nitride to conduct electricity at all. This difference is due to that it requires fairly little energy to get the electrons in graphene moving, while the electrons in boron nitride need a lot of extra energy to get to a state where they are mobile. This energy boost that is needed for the electrons to be able to move is also a measure of the band gap (corresponding to a gap in energy between different states that the electrons can be in). Graphene thus has an extremely small band gap, boron nitride has a large bandgap. By combining the two materials people want to create a hybrid material with a band gap of a size that is useful for e.g. applications in electronics.

However, it turns out that you cannot just replace a few carbon atoms with boron and nitrogen. How the boron and nitrogen atoms are arranged in relation to each other matters for how large the band gap of the resulting material turns out to be. What the American research group has done is try to predict the size of the bandgap based on the placement of boron and nitrogen atoms using artificial neural networks, more specifically so-called CNNs or convolutional neural networks.

The neural networks

CNNs are a type of neural networks that have been developed to pick out characteristic features from images and then classify the images based on the features - they are for example useful for facial recognition and when self-driving cars need to tell the difference between a pedestrian and a road sign. The basic principle of a CNN is similar to comparing small regions of a picture with smaller, simpler images and give a positive response if they are similar. If for example you have a picture of a house and the smaller image has a vertical line you might get a positive response when you get to the corners, doors or windows since their depictions contain straight, vertical sections. In  CNN, however, you have to represent both images as matrices of numbers, and you also have several layers where the result of one comparison to a smaller image in turn is compared with more matrices (this is needed for the identification of more complex features in the image). Also note that the smaller image (or filter) is not something you define beforehand, but something the network learns. If your training data contains no straight lines, the filters that result from training probably will not do so either.



To be able to use CNNs for the graphene problem described above the researchers chose to use computer models where each pair of atoms is represented by numbers. When they introduce boron and nitrogen atoms in graphene they usually come in pairs, with a boron and a nitrogen atom next to each other. The researchers therefore chose to represent a boron-nitrogen pair with a "one" and a carbon-carbon pair with a "zero", and thereby constructed an image of the material that different types of CNNs can handle. They also built their networks to give the size of the band gap as output data.

Neural networks need to be trained with relevant data in order to work, something that usually involves automatically comparing the output of hte network to the desired result, calculating the deviation, and adjusting the neural network to give a better answer. In order to train their neural networks the researchers therefore generated several thousand possible configurations and calculated the bandgap of each configuration using density functional theory. The trained networks were then used to predict the bandgap for another batch of configurations where the calculated bandgap was known but that were not used in training. The results turned out to be very promising.

What can we learn from this?

So what is the effect of this study? The researchers have successfully shown that it is possible to predict certain properties of materials using neural networks, which should give those who do research on graphene and other two-dimensional materials another tool that they can use in their research. There is still a long way to go from this particular study to electronics based on graphene and boron nitride, but it may make it easier to know what kind of material configurations are worth working on.

Another interesting thing about this study is what it says between the lines about the limits of machine learning. For the method to work at all the neural network needs to receive all the relevant information in a format it can process, which means that it requires quite a bit of knowledge about graphene and boron nitride to even formulate the problem in such a way that it can be tackled. For example, in this study the researchers have chosen to focus completely on where each boron-nitrogen pair is situated in relation to other pairs and thus discarded all other characteristics of the material, presumably based on what is already known about these materials. (As a example, the relative orientation of neighbouring boron-nitrogen pairs is completely ignored - is it boron-nitrogen-nitrogen-boron or boron-nitrogen-boron-nitrogen? This information has been trimmed away before the neural network is involved.)

A known limitation of neural network is that it is hard to understand why they work the way they do even when they give good results. In a study like this one it would have been very interesting to see what the structures with low or high bandgap respectively have in common, but that is not information that is easy to extract from the neural network itself and the researchers seem not to have made any effort to try. I strongly suspect that a method to understand what is going on inside the networks is necessary for this type of study to be helpful in understanding the studied materials.

As you have probably understood by now, I do not quite agree with Steven Novella about this one fairly limited study showing that neural networks will do decades of research in a few days and take us significantly closer to graphene electronics, but the results in it are still interesting as an example of machine learning in materials physics.