Development of a policy uncertainty metric using large language models

Rising policy uncertainty increases the tendency to delay decisions and to wait and see

In March 2024, the Bank of Japan ended its negative interest rate policy and raised interest rates for the first time in 17 years. In April 2025, President Trump announced that the U.S. will impose a 24% reciprocal tariff on Japan. The Japanese government had set a goal of achieving a surplus in the primary balance in fiscal 2025, but this goal is unlikely to be met.

As the economy globalizes, there are more situations where the policies of other countries determine the economy of the home country. Until the 1990s, when the world was heading toward economic growth, the national policies of developed countries often followed a similar direction. However, since the global financial crisis of 2008, countries have more frequently tried to protect themselves from losses. Managing policy uncertainty is becoming increasingly important.

Policy uncertainty refers to the lack of clarity concerning the future of the economy, which largely increases in relation to policy. People put in a different environment often react more radically than policy makers expect. The metrics created to quantify this are policy uncertainty indices. They also serve as metrics that comprehensively capture the effects that the policies will generate.

When policy uncertainty increases and the future becomes unpredictable, more companies delay their decisions, such as capital investments and hiring, and wait for the situation to become clearer. In some cases, this can lead to a positive outcome, but it can also result in a missed opportunity. If employment is curtailed, it will directly impact our lives. In addition, the precautionary savings effect, in which households and companies hold cash, increases.

In this way, policy uncertainty is closely related to economic activity and serves as an important metric for measuring economic activity, so it is desirable to measure it accurately. However, policies are interrelated. When the Bank of Japan buys Japanese government bonds for monetary policies, fiscal policies will then be a factor in how to repay them from taxes. Since all policies are inseparable, it is necessary to consider the whole picture besides looking at individual policies, which makes measurement difficult.

Policy uncertainty is typically measured using newspaper articles containing relevant keywords

There are several ways to measure policy uncertainty, but a common method is to measure it by the number of newspaper articles containing relevant keywords. An American scholar first published it in a paper in 2016, and various countries began to create such metrics.

In Japan, the Research Institute of Economy, Trade and Industry (RIETI) publishes its policy uncertainty index every month. In this measurement, articles published in four major newspapers that contain at least one predetermined keyword are collected monthly for each newspaper. These keywords are classified into categories such as economy, policy, and uncertainty. For example, “economy” and “economic condition” are in the economy category, “taxation” and “taxation system” are in the policy category, and “uncertain” and “unclear” are in the uncertainty category.

In addition, the presence of other predetermined keywords is used to judge which policy area the collected articles relate to, such as fiscal policy, monetary policy, exchange rate policy, or trade policy. If the article contains at least one of the keywords “financial resources,” “taxation system,” “expenditures,” “social security benefits,” or “national debt,” it is considered to be related to fiscal policy.

Articles are retrieved using these keywords, and the policy uncertainty index is calculated from the data on the number of such articles. At present, the level is adjusted so that the average value for the period from January 1987 to December 2015 is 100. Uncertainty is considered increased when the index exceeds 100. Looking at the graph for the 30 years for which RIETI has released data, it can be seen that since the global financial crisis of 2008, the fluctuation of policy uncertainty has become significantly wider.

However, measurement using keywords also has its weaknesses. For instance, even if an article conveys a sense of policy uncertainty, it is not counted unless it contains a keyword. The measurement also cannot distinguish whether the lack of clarity mentioned in the content refers to the effects of past policies or future policies. Because it assesses by keywords, it does not go into detail.

Measurement using LLMs that distinguishes between past and future, home and other countries

To overcome the limitations of measurement using keywords, our research team is developing a method to extract the presence and type of uncertainty from the content of articles using LLMs.

Our study first focused on monetary policy. There are at least two types of monetary policy uncertainty. One is the uncertainty regarding future monetary policy, and the other is the uncertainty regarding the effects of monetary policy taken so far. However, the keyword-based judgment does not distinguish them. So we analyzed years 2015 and 2016, considering that the negative interest rate policy was introduced in January 2016.

We focused on this period because the 2% inflation rate aimed for at the beginning of 2015 was not achieved and articles indicating past and future uncertainties were mixed together. Both types of articles contained the same keywords and were not isolated.

Three team members read the articles to determine whether they were about the past or the future based on context, and then discussed and sorted them. Our uniqueness is that we have made it possible to separate the policy of the home country from that of a foreign country. With the goal of using articles referring to the home country and future as a metric, we created a training set, or so-called labeled data, to enable categorization of past and future, home and other countries based on the understanding of the context.

Simply put, labeled data is the data used to train LLMs how to read articles like these. It can be considered a source material that will ultimately allow LLMs to understand context from the same perspective. We allowed an LLM to learn from about 200 sets created in this manner and then used the fine-tuned model to classify all articles.

This revealed that whether past or future uncertainty had a stronger impact depended on the period. At the beginning of 2016, many articles showed uncertainty about past policies, but from around September 2016, uncertainty about future policies began to increase. We will conduct an empirical analysis to determine whether this index is a variable that significantly explains changes in economic or financial variables, such as GDP or the exchange rate.

We would like to create more sophisticated metrics to show future policy uncertainty. Currently, creating labeled data is a daunting task, but if we can develop a simpler method to distinguish data, we should be able to provide information in real time. Reducing uncertainty makes it easier for people and companies to plan for the future. In addition, when uncertainty increases, the government needs to communicate its policies more thoroughly to its citizens, so it should also serve as a means of raising awareness of this issue. I hope that we can contribute to society by measuring policy uncertainty more accurately.

* The information contained herein is current as of March 2025.
* The contents of articles on Meiji.net are based on the personal ideas and opinions of the author and do not indicate the official opinion of Meiji University.
* I work to achieve SDGs related to the educational and research themes that I am currently engaged in.

Information noted in the articles and videos, such as positions and affiliations, are current at the time of production.