Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since limited data is a common problem in tourism time-series forecasting, we perform a comprehensive comparison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point out necessary future improvements.

The evidence to be evaluated is:Most of these studies make predictions with only a limited set of models.

Many studies focus on long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions as city or country-level tourism demand Asvikarani et al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.

Most of these studies make predictions with a limited set of models.

However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control. Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts. The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria.

This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinct frequency of recording.

For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is a key challenge for this real-world prediction task.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search conducting three runs per configuration and keeping the one which achieved the lowest RMSE.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001).

There are several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series to try and minimize the deviation between them

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs. For the DL models, we added additional features to the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a One-Hot-Encoded single word description of the weather (e.g. "Snow"). We performed further pre-processing by normalizing all features to values between \(0\) and \(1\). To account for seasons, we performed sine-cosine transformation for the month.

Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.

Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fitting. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training.

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which is the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's Birthplace Museum around the time of New Year's Eve.

Finally, on the bottom the predictions for the Festival Hall Festspilehaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

We conducted a second set of experiments on the timeframe from 2019 to 2021 since only for this period geolocation data of individual tourists was available.

This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons. Since the _Salzburg Card dataset_ for this particular timeframe contains a significantly lower number of datapoints due to lockdowns enforced by the government, the numbers must not be compared directly to the results discussed in the last section. Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach. **Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions.** Since there is more sparse geolocation data expected to be processed within real-life-scenarios, this is the only approach to fit these needs.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error. This work therefore serves as a proof-of-work for incorporating spatial structure using GNNs that is robust to overfitting even when fed sparse data and prediction accuracy should improve a lot when denser geolocation data is available.

In addition, we want to use the knowledge gained to build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data.

In order to ensure reproducibility we publish the _Salzburg Card dataset_ alongside this paper.

The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH.

We accumulated the hourly amount of entries into these points of interest(POI's). Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves as a baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic.

Not all visitors need to use this card so this data represents a subset of all tourists visiting these POI's. \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-stated. The study does present a comprehensive comparison between DNNs and traditional techniques (specifically ARIMA) on a limited dataset, as demonstrated by:
1. Clear documentation of dataset limitations (3% tourist coverage, sparse data)
2. Systematic evaluation across multiple models (7 DNN variants, ARIMA)
3. Detailed performance metrics (MAE, RMSE) and runtime comparisons
4. Analysis of model behavior across different scenarios (different POIs, sparse vs dense data)
5. Transparent discussion of limitations and future improvements needed

The evidence thoroughly supports the claim's assertion about performing a comprehensive comparison and revealing shortcomings on a small dataset.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since limited data is a common problem in tourism time-series forecasting, we perform a comprehensive comparison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point out necessary future improvements.

The review comment to be evaluated is:

# Strengths
* The paper addresses an interesting spatiotemporal prediction problem.
* The paper attempts the important work of benchmarking a diverse collection of methods on a difficult task.
* The dataset which this paper will publicly release will no doubt be useful to the community, especially since it has data for before and during COVID.
* Investigating the use of side information (sparse geo-location information from tourists and metadata about weather and holidays) is worthwhile and interesting.

# Weaknesses
* The paper trains 7 different deep learning approaches with the same learning rate and reports the results. I believe it is uncontroversial to claim that tuning the learning rate is perhaps the most important step in training a deep learning algorithm. Given this, it is difficult to trust the results in the paper.
* It is not clear whether a validation set was used or (if a validation set was used) how it was constructed. Also, which split is used for model selection?
* The ARIMA model gets to use the true values for past time points before making a prediction. Do the deep learning models get a similar advantage? If so, how is it implemented?
* The results are not contextualized. At what point is an improvement meaningful? Are MAE and RMSE effectively capturing what is important for the end-user? For instance, in Table 1 we see that GRU-D is better than CT-RNN by 0.01 RMSE when doing "only visitors" prediction. Is this a meaningful difference for the use case? Does this represent a stable difference between the models, or would the results reverse if we trained again?
* There is no discussion of the potential for bias in the data. Is there a good reason to believe that Salzburg Card holders are a representative sample of the population of tourists? How is the mobile phone data sampled from the population? If these datasets are biased, what effects could they have on the models?

The evidence to be evaluated is:Most of these studies make predictions with only a limited set of models.

Many studies focus on long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions as city or country-level tourism demand Asvikarani et al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.

Most of these studies make predictions with a limited set of models.

However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control. Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts. The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria.

This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinct frequency of recording.

For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is a key challenge for this real-world prediction task.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search conducting three runs per configuration and keeping the one which achieved the lowest RMSE.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001).

There are several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series to try and minimize the deviation between them

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs. For the DL models, we added additional features to the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a One-Hot-Encoded single word description of the weather (e.g. "Snow"). We performed further pre-processing by normalizing all features to values between \(0\) and \(1\). To account for seasons, we performed sine-cosine transformation for the month.

Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.

Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fitting. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training.

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which is the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's Birthplace Museum around the time of New Year's Eve.

Finally, on the bottom the predictions for the Festival Hall Festspilehaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

We conducted a second set of experiments on the timeframe from 2019 to 2021 since only for this period geolocation data of individual tourists was available.

This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons. Since the _Salzburg Card dataset_ for this particular timeframe contains a significantly lower number of datapoints due to lockdowns enforced by the government, the numbers must not be compared directly to the results discussed in the last section. Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach. **Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions.** Since there is more sparse geolocation data expected to be processed within real-life-scenarios, this is the only approach to fit these needs.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error. This work therefore serves as a proof-of-work for incorporating spatial structure using GNNs that is robust to overfitting even when fed sparse data and prediction accuracy should improve a lot when denser geolocation data is available.

In addition, we want to use the knowledge gained to build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data.

In order to ensure reproducibility we publish the _Salzburg Card dataset_ alongside this paper.

The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH.

We accumulated the hourly amount of entries into these points of interest(POI's). Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves as a baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic.

Not all visitors need to use this card so this data represents a subset of all tourists visiting these POI's. \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim appears overstated relative to the available evidence. While the study does compare DNNs and traditional techniques (ARIMA) on a tourism dataset, several critical limitations undermine the "comprehensive" nature of the comparison:

1. As noted in the review comments, all DL models used the same learning rate without proper tuning
2. Validation methodology is unclear
3. Limited model selection process explanation
4. The study uses only one traditional technique (ARIMA) rather than a comprehensive set
5. Dataset represents only ~3% of tourists through Salzburg Card data
6. Lack of statistical significance testing for model comparisons
7. No discussion of potential data biases

While valuable comparisons were made, characterizing it as "comprehensive" overstates the scope and rigor of the analysis.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since limited data is a common problem in tourism time-series forecasting, we perform a comprehensive comparison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point out necessary future improvements.

The review comment to be evaluated is:

Strength:

The prediction performance is compared on several deep learning methods on continuous sequential predictions.

Including external features to the times series dataset could slightly improve the DL models’ performance.

Weakness:

The authors mentioned “limited data is a common problem”; but no solution was discussed to overcome this challenge. Instead, Only supervised learning models were applied for training and prediction.

When comparing the model performance on time-series prediction problem, it would be great to further compare with the naïve prediction by simply setting all forecasts to be the value of the last observations, i.e., X_t = X_{t-1}.

In general, this work shows a comprehensive comparison across different SOTA models; however, the contribution on creating new models/approaches and theoretical insight is still missing.

Some contents in the introduction and related work look redundant.

The evidence to be evaluated is:Most of these studies make predictions with only a limited set of models.

Many studies focus on long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions as city or country-level tourism demand Asvikarani et al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.

Most of these studies make predictions with a limited set of models.

However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control. Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts. The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria.

This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinct frequency of recording.

For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is a key challenge for this real-world prediction task.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search conducting three runs per configuration and keeping the one which achieved the lowest RMSE.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001).

There are several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series to try and minimize the deviation between them

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs. For the DL models, we added additional features to the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a One-Hot-Encoded single word description of the weather (e.g. "Snow"). We performed further pre-processing by normalizing all features to values between \(0\) and \(1\). To account for seasons, we performed sine-cosine transformation for the month.

Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.

Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fitting. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training.

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which is the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's Birthplace Museum around the time of New Year's Eve.

Finally, on the bottom the predictions for the Festival Hall Festspilehaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

We conducted a second set of experiments on the timeframe from 2019 to 2021 since only for this period geolocation data of individual tourists was available.

This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons. Since the _Salzburg Card dataset_ for this particular timeframe contains a significantly lower number of datapoints due to lockdowns enforced by the government, the numbers must not be compared directly to the results discussed in the last section. Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach. **Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions.** Since there is more sparse geolocation data expected to be processed within real-life-scenarios, this is the only approach to fit these needs.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error. This work therefore serves as a proof-of-work for incorporating spatial structure using GNNs that is robust to overfitting even when fed sparse data and prediction accuracy should improve a lot when denser geolocation data is available.

In addition, we want to use the knowledge gained to build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data.

In order to ensure reproducibility we publish the _Salzburg Card dataset_ alongside this paper.

The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH.

We accumulated the hourly amount of entries into these points of interest(POI's). Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves as a baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic.

Not all visitors need to use this card so this data represents a subset of all tourists visiting these POI's. \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the authors did perform a comprehensive comparison between DNNs and traditional techniques (ARIMA) on a small dataset of Salzburg Card data from 2017-2019. They clearly demonstrated limitations such as:
- Limited training data affecting feature utilization
- Overfitting issues with sparse data
- Model-specific shortcomings (e.g., ARIMA's inability to handle additional features)
- Performance tradeoffs between different approaches

The claim is well-stated as it accurately reflects what was actually done in the study without exaggerating the findings. The evidence supports both the comprehensive nature of the comparison and reveals genuine shortcomings that inform future work.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since limited data is a common problem in tourism time-series forecasting, we perform a comprehensive comparison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point out necessary future improvements.

The review comment to be evaluated is:

Strengths:
- the problem is interesting and worth investigating
- the data is released and could be useful for the community
- several methods were tested

Weaknesses:
- the paper does not propose a methodological novelty
- the problem is in the end very simple (single time series forecasting one step ahead), if we do not consider the geolocalization data with graphs (which currently does not work).
- use of MAE as the loss: this is unusual as MAE is not differentiable, so usually we use MSE.. why and how did you use MAE?
- the final goal is not clear, as the long introduction mixes Covid problems, increase of tourists problems (which are both unrelated), ethical problems due to goelocalizations bought to have more data.
- the paper does not seem to be novel with respect to state-of-the-art in this specific field, and experiments are not clear. Please state which are methods taken from previous works, and what is your method that would be novel. Is the statistical baseline, ARIMA, used in practice in this application? What is the gold standard today in tourist forecast?
- the forecast is only one step ahead, i.e. 1 hour, and it would be more interesting to augment this. Also, please indicate more clearly (i.e. from the start) what is the target (prediction of next hour entries).

The evidence to be evaluated is:Most of these studies make predictions with only a limited set of models.

Many studies focus on long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions as city or country-level tourism demand Asvikarani et al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.

Most of these studies make predictions with a limited set of models.

However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control. Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts. The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria.

This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinct frequency of recording.

For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is a key challenge for this real-world prediction task.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search conducting three runs per configuration and keeping the one which achieved the lowest RMSE.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001).

There are several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series to try and minimize the deviation between them

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs. For the DL models, we added additional features to the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a One-Hot-Encoded single word description of the weather (e.g. "Snow"). We performed further pre-processing by normalizing all features to values between \(0\) and \(1\). To account for seasons, we performed sine-cosine transformation for the month.

Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.

Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fitting. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training.

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which is the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's Birthplace Museum around the time of New Year's Eve.

Finally, on the bottom the predictions for the Festival Hall Festspilehaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

We conducted a second set of experiments on the timeframe from 2019 to 2021 since only for this period geolocation data of individual tourists was available.

This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons. Since the _Salzburg Card dataset_ for this particular timeframe contains a significantly lower number of datapoints due to lockdowns enforced by the government, the numbers must not be compared directly to the results discussed in the last section. Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach. **Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions.** Since there is more sparse geolocation data expected to be processed within real-life-scenarios, this is the only approach to fit these needs.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error. This work therefore serves as a proof-of-work for incorporating spatial structure using GNNs that is robust to overfitting even when fed sparse data and prediction accuracy should improve a lot when denser geolocation data is available.

In addition, we want to use the knowledge gained to build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data.

In order to ensure reproducibility we publish the _Salzburg Card dataset_ alongside this paper.

The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH.

We accumulated the hourly amount of entries into these points of interest(POI's). Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves as a baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic.

Not all visitors need to use this card so this data represents a subset of all tourists visiting these POI's. \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about performing a "comprehensive comparison" on a small dataset appears partially overstated. While the study does compare multiple models (ARIMA, various DNNs, Transformers, GNNs) and includes different data features (visitor counts, weather, holidays), several limitations exist:

1. The reviewer points out the simplicity of the problem (single time series, one-step-ahead prediction)
2. The evidence shows limited exploration of ARIMA capabilities (non-seasonal variant only)
3. The reviewer notes lack of clarity about state-of-the-art comparisons and gold standards
4. The small dataset limitation is acknowledged but not thoroughly analyzed in terms of its impact
5. The study does not clearly establish what constitutes "comprehensive" in this domain

While some comparison is performed, characterizing it as "comprehensive" overstates the scope and depth of the analysis.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since limited data is a common problem in tourism time-series forecasting, we perform a comprehensive comparison of the DNNs and traditional techniques on a small dataset to reveal the shortcomings and point out necessary future improvements.

The review comment to be evaluated is:

Strengths:

1. The description of the problem is clear.

2. This paper is the first to apply deep learning methods to the tourist flow prediction problem.

Weaknesses:

1. The innovation of this paper is insufficient. Despite experiments that have been carried out on several methods, the authors do not propose their own methods.

2. The authors do not give a detailed explanation of why GNN methods seem to outperform other methods.

3. The experiment part in this paper is not enough. It lacks an ablation study part and a case study part.

The evidence to be evaluated is:Most of these studies make predictions with only a limited set of models.

Many studies focus on long-term predictions of monthly, quarterly and yearly, or in the best case daily number of visitors Bi et al. (2020) in large regions as city or country-level tourism demand Asvikarani et al. (2020). However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control.

Most of these studies make predictions with a limited set of models.

However, performing granular predictions on an hourly basis and per POI is critical for the task of tourism flow control. Two different data sources were combined to enable the use of their different features in the training of the models and prediction of future visitor counts. The first dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH. Upon purchase of these cards, the owner has the ability to enter 32 different tourist attractions and museums included in the portfolio of the Salzburg Card. The dataset consists of the time-stamps of entries to each POI. Additionally, we used data about weather and holidays in Austria.

This dataset roughly covers app. 3% of tourists and helps us in better evaluating tourists inbetween POI's. This dataset relies on location data transmitted by mobile-phone apps and contains sparse data about tourist-locations without any destinct frequency of recording.

For this work, we built our own dataset on hourly data collected from tourist attractions and then expanded this by including geolocation data. Including many different datasources is a key challenge for this real-world prediction task.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In order to find optimal model size, loss function, and whether to use normalized visitor counts, we did a grid search conducting three runs per configuration and keeping the one which achieved the lowest RMSE.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001).

There are several recommended approaches for the manual selection of parameters and they all rely on comparing the ARIMA model fit to the real values of the time-series to try and minimize the deviation between them

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

The data then consists of the hour of the day, and the number of entries at that hour to each of the 32 POIs. For the DL models, we added additional features to the dataset: Year, Month, Day of month, Day of week, Holidays and Weather data. For the Holiday data we used the national holidays and school holidays and count the days to the next school day. For the Weather data, we use the hourly weather data with these features: Temperature, Feels Like, Wind speed, Precipitation, and Clouds as well as a One-Hot-Encoded single word description of the weather (e.g. "Snow"). We performed further pre-processing by normalizing all features to values between \(0\) and \(1\). To account for seasons, we performed sine-cosine transformation for the month.

Finally, We split the data into sequences of length 30, and used the data from years 2017 and 2018 as the training set, and 2019 as the test set.

Our dataset of tourist locations was very sparse which subsequently resulted in very sparse inputs for each node.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features. Providing the additional features to the models resulted in a slightly better performance for the DL models. The improvements were quite marginal because we have limited training data available and thus increasing the number of features does not pay off, or even might result in over-fitting. ARIMA was not able to use the external features and fails when given only short sequences as input. The DL models can handle shorter sequences when trained on the whole training set.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training.

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time. Thus, ARIMA loses the implicit data about the total number of visitors in the city.

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models. The plot on the Top shows the forecast and real values for the tourists entered the Funicular Railway descend which is the cable car ride leading up to Salzburg Castle. As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately. Mid shows visitor predictions for Mozart's Birthplace Museum around the time of New Year's Eve.

Finally, on the bottom the predictions for the Festival Hall Festspilehaus guided tour are shown which is sparse since it takes place once a day at 2 pm. All models fail in prediction for the second and third peak at this location. However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

We conducted a second set of experiments on the timeframe from 2019 to 2021 since only for this period geolocation data of individual tourists was available.

This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons. Since the _Salzburg Card dataset_ for this particular timeframe contains a significantly lower number of datapoints due to lockdowns enforced by the government, the numbers must not be compared directly to the results discussed in the last section. Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison. However, all other methods failed to converge when trained on the sparse geolocation data which shows the usefulness of the GNN approach. **Only with GNNs we are able to utilize sparse geolocation Data into meaningful predictions.** Since there is more sparse geolocation data expected to be processed within real-life-scenarios, this is the only approach to fit these needs.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error. This work therefore serves as a proof-of-work for incorporating spatial structure using GNNs that is robust to overfitting even when fed sparse data and prediction accuracy should improve a lot when denser geolocation data is available.

In addition, we want to use the knowledge gained to build specialised models which outperform state-of-the art models in terms of short-term prediction with limited data.

In order to ensure reproducibility we publish the _Salzburg Card dataset_ alongside this paper.

The dataset we used stems from the "Salzburg Card" which was kindly provided to us by TSG Tourismus Salzburg GmbH.

We accumulated the hourly amount of entries into these points of interest(POI's). Every entry from 2017 to 2021 using thses cards was recorded and summed up in this dataset. 2017 to 2019 serves as a baseline of "normal" tourism activity in the city of Salzburg. The data from 2020 and 2021 is of particular interest since the visitor numbers were heavily influenced by the global CoVID-19 pandemic.

Not all visitors need to use this card so this data represents a subset of all tourists visiting these POI's. \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I assess that the claim is partially overstated. While the authors did perform comparative analysis between DNNs and ARIMA on a relatively small dataset (2017-2019 Salzburg Card data), several aspects raise concerns:

1. The comparison wasn't truly "comprehensive" - ARIMA was excluded from later experiments due to computational constraints
2. The analysis of shortcomings focuses mainly on ARIMA's limitations rather than DNN weaknesses
3. Review comments note lack of ablation studies and insufficient experimental depth
4. The "necessary future improvements" mentioned in the claim aren't clearly detailed in the evidence beyond a brief mention of building "specialized models"

However, the authors did provide detailed performance metrics, timing comparisons, and visual analyses of model behaviors, showing a legitimate comparative effort within the scope of their data.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this paper, we empirically evaluate the performance of state-of-the-art deep-learning methods such as RNNs, GNNs, and Transformers as well as the classic statistical ARIMA method.

The evidence to be evaluated is:Existing work examines tourist demand forecasting using Recurrent Neural Networks such as LSTMs Rizal & Harati (2016); Li & Cao (2018) or hidden Markov Models in conjunction with deep neural networks Yao & Cao (2020). Most of these studies make predictions with only a limited set of models.

Existing work examines tourist demand forecasting using Recurrent Neural Networks such as LSTMsRizal & Hartati (2016); Li & Cao (2018) or hidden Markov Models in conjunction with deep neural networks Yao & Cao (2020). Most of these studies make predictions with a limited set of models.

We then perform predictions with a rich set of models and do a comprehensive comparison of the results.

Then we go over the methods we chose to evaluate and compare their performances. We use a large set of RNN variations on the tourist-flow dataset to perform a comprehensive comparison of the state-of-the art models and provide insight on their performance. The set comprises vanilla-RNN, LSTM, phased-LSTM, GRU-D, CT-RNN, CT-LSTM and Neural-ODE networks. Moreover, we used a Transformer model, using only the encoder part with 8 heads, 64 hidden units, and 3 layers, to forecast the tourist flow. Finally, we applied Attention Temporal Graph Convolutional Networks (A3T-GCN) Zhu et al. to our prediction problem in order to utilize geolocation data of individual tourists.

In this study we use the non-seasonal variant of the ARIMA model which does not consider the seasonal patterns in a time-series. This model is usually denoted as _ARIMA (\(p\),\(d\),\(q\))_, where \(p\) is the number of autoregressive terms, \(d\) is the number of non-seasonal differences, and \(q\) is the number of lagged forecast errors in the prediction equation Burger et al. (2001). A generic expression for the non-seasonal ARIMA process is given as Hyndman & Khandakar (2008): \[\phi(B)(1-B)^{d}y_{t}=c+\theta(B)\varepsilon_{t}\]

For this study, we have used the _auto.arima_ function from the _R forecast_ library that automatically fits the ARIMA model with different sets of parameters and returns the best combination of \(p\), \(d\), and \(q\) for the given time-series Hyndman & Khandakar (2008); Hyndman & Athanasopoulos (2018). Given that each of the 32 POIs in this study has a different time-series of visitor counts, we have determined different ARIMA parameters for each POI which were the best fit for its time-series. ARIMA predictions models were then build individually for each POI by fitting the ARIMA to that POI's training dataset with the best \(p\), \(d\), and \(q\) parameters for that POI using the Python _pmdarima_ library.

We used the Salzburg card data from years 2017, 2018, and 2019 for our first set of experiments.

Our results show partial success for the GNN approach since it was able to handle the very sparse tracking data which led to overfitting in all other models. We performed a diverse set of experiments with ARIMA and DL models to evaluate their forecasting accuracy, execution time and prediction time and compare the models. Table 1 shows the Mean-Absolute-Error (MAE) and Root-Mean-Squared-Error (RMSE) achieved for each method applied to the timeframe from 2017-2019 - before COVID. For the deep-learning models model size, corresponding number of parameters and training and prediction times of the best run are listed additionally.

All of our deep-learning models were able to outperform the ARIMA method in both metrics, with and without additional features.

ARIMA was not able to use the external features and fails when given only short sequences as input.

We measured training times for the DL models and prediction times for all models on a standard workstation machine setup (4 CPU cores with 3.9 GHz, 8GB DDR3 memory) equipped with a single GPU (GeForce GTX 1050 Ti, 4GB memory). As shown in Table 1 ARIMA took 69s to perform a single prediction for all POIs and the DL took for the same task fractions of milliseconds, while having used once y minutes for training. It is not possible to directly compare training and testing runtimes between them because ARIMA does not have a dedicated training step where the model is built. Instead, before every new prediction at time t, ARIMA model is fitted to the known time series up until the time t-1 (i.e., training and testing are integrated for every prediction).

ARIMA calculations are time-suming, since it makes predictions for each POI separately, while the DL models are trained with the visitors to all POIs in a single vector and make predictions for all at the same time.

In order to visually explore the predictions made by the models, we plotted the predictions and the ground truth for a few selected time-windows (see Figure 2).

Our plots show that although ARIMA is out-performed by the DL methods in the average error of all predictions, there are cases where it actually performs better than the other models.

As visible in the plot, the DL models show a better performance, especially in the valleys where the ARIMA fails to predict the downfalls accurately.

However, CT-RNN shows a very good performance in predicting the first and last peak and at least shows an upward trend for the second and third peak.

Results are presented in Table 2 which shows for each model the number of Parameters and MAE with and without additional features andalso when using features and the geolocation data. This time we included the Transformer and GNN models, but excluded ARIMA and ANODE due to computation time reasons.

Relatively speaking, the RNN models show comparable prediction accuracy although Transformers led to the best results. Transformers can handle multi-variate data well due to the multi-head self-attention mechanism which enables them to extract hidden correlations in input, and hence get better loss after using additional features. However, they require considerably more parameters in comparison to the RNN models. For GNNs we only conducted experiments with additional geolocation data since input graph attributes would be even sparser defeating the point of using a graph based approach. The A3T-GCN algorithm scored a slightly worse prediction error in comparison.

We performed a thorough evaluation of deep-learning (DL) versus ARIMA, a traditional method, on the task of forecasting tourist flow time-series. We found that all of the DL models were able to outperform the established ARIMA method when using only visitor counts as the training data. Extending the dataset with additional features of time, holidays, and weather improved the predictions of the DL models, while ARIMA is not able to handle additional features.

In terms of the prediction time, DL models are meaningfully faster than ARIMA, which is an important aspect since most real-world applications require fast inference time. We then evaluated various DL techniques including Transformer and GNN on the timeframe where COVID hit while also incorporating sparse tracking data from mobile phones.

Hoewever, when adding the geolocation data, all of the RNN models including the Transformer diverged. The A3T-GCN model on the other hand, was able to deal with this additional data albeit resulting in a slightly higher validation error.

The most straightforward next step is to try to improve the performance of the DL methods using additional training techniques such as regularization or learning rate scheduling. Admittedly, the ARIMA method is most suitable for predicting univariate data which is why our classical methods should be extended by also considering methods for multivariate forecasting such as Vector autoregression (VAR). \begin{table}
\begin{tabular}{l c c c|c c|c c} \hline \hline  & \# Cells / & \multicolumn{2}{c|}{Time} & \multicolumn{2}{c|}{only visitors} & \multicolumn{2}{c}{external features} \\
**Model** & \# Parameters & Train (min) & Pred (ms) & MAE & RMSE & MAE & RMSE \\ \hline ARIMA & **224** & - & 69k & 5.217 & 7.833 & - & - \\ \hline ANODE & 64 / 21.3k & 145.6 & 3.01 & 4.599 & 6.965 & 4.410 & 6.663 \\ Vanilla RNN & 128 / 43.7k & 5.9 & **0.18** & 3.958 & 6.321 & 3.802 & 6.160 \\ LSTM & 32 / 11.9k & **1.5** & 0.24 & 3.713 & 6.209 & 3.630 & 6.113 \\ Phased LSTM & 32 / 11.8k & 27.0 & 0.46 & 3.825 & 6.359 & 3.651 & 6.120 \\ CT-LSTM & 32 / 19.9k & 18.1 & 0.31 & 3.734 & 6.239 & 3.700 & 6.185 \\ CT-RNN & 128 / 27.4k & 57.1 & 0.60 & 3.694 & 6.131 & 3.629 & **5.983** \\ GRU-D & 64 / 27.7k & 16.6 & 0.33 & **3.638** & **6.121** & **3.621** & 6.073 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

 \begin{table}
\begin{tabular}{l c|c|c|c} \hline  & & \multicolumn{3}{c}{MAE} \\
**Model** & \# Parameters & only visitors & + features & + additional geolocation \\ \hline Vanilla RNN & 11.0k & 2.75 & 2.58 & 430.88 - diverging \\ LSTM & 23.6k & 2.49 & 2.41 & 422.63 - diverging \\ Phased LSTM & 23.4k & 2.56 & 2.40 & 426.89 - diverging \\ CT-LSTM & 39.9k & 2.51 & 2.40 & 428.47 - diverging \\ CT-RNN & **7.0k** & 2.54 & 2.48 & 431.74 - diverging \\ GRU-D & 27.7k & 2.53 & 2.46 & 431.71 - diverging \\ Transformer & 253k & **2.35** & **2.23** & 482.78 - diverging \\ \hline A3T-GCN & 39.7k & - & - & **2.71** \\ \hline \end{tabular}
\end{table}
Table 2: The prediction results for 2021 after training on data from 2019 & 2020. Each model (except the GNN) was trained on just the visitors, visitors and external features and visitors with both features and tracking data. Reported is the best MAE on the validation set for three different initializations. Using the sparse tracking data led to divergence in all models except GNNs.

Input: Figure 2: Predicted and True visitor counts for the Funicular Railway (top) and Mozart’s Birthplace Museum (mid) and the Festival Hall (bottom). Predictions are computed using CT-RNN (orange), GRU-D (green) and ARIMA (red).

Input: Table 2: The prediction results for 2021 after training on data from 2019 & 2020. Each model (except the GNN) was trained on just the visitors, visitors and external features and visitors with both features and tracking data. Reported is the best MAE on the validation set for three different initializations. Using the sparse tracking data led to divergence in all models except GNNs.

Input: Table 1: The prediction results for models when using only the visitors count data and when using additional features from external data. Our experimental results show that the errors were smaller for RNNs compared to ARIMA both with and without additional features were used.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement appears relatively well-stated but has some minor overstatements. The evidence shows comprehensive testing of RNNs (multiple variants), Transformers, and GNNs, as well as ARIMA, validating most of the claim. However, calling all methods "state-of-the-art" may be slightly overstated, as some are more standard approaches. The empirical evaluation is well-documented with detailed experiments, performance metrics, and comparative analysis across different settings. The evidence includes thorough experimental results presented in tables and figures that support the systematic evaluation claimed.

<score>2</score>

Justification:
1. The evidence demonstrates actual empirical evaluation of multiple model types as claimed
2. Tables 1 and 2 provide comprehensive performance metrics
3. Experiments cover different scenarios (with/without features, with geolocation)
4. The term "state-of-the-art" is slightly overstated as some methods are standard
5. Clear experimental details are provided for reproducibility
6. Results include both quantitative metrics and visual analysis (Figure 2)
7. Direct comparisons between methods are shown in terms of accuracy and computational efficiency