1984-1990 | 0.67 | 0.01 | 0.68 | (0.00) | (0.00) | (0.00)
(a) Transpacific (encouraging small firms)

Regime | CS | PS | SW | CS change | PS change | SW change
---|---|---|---|---|---|---
1973-1979 | 2.12 | 0.10 | 2.22 | (0.00) | (0.00) | (0.00)
1980-1983 | 0.64 | 0.02 | 0.66 | (0.00) | (0.00) | (0.00)
1984-1990 | 0.67 | 0.01 | 0.68 | (0.00) | (0.00) | (0.00)
(b) Transpacific (encouraging large firms)

Regime | CS | PS | SW | CS change | PS change | SW change
---|---|---|---|---|---|---
1973-1979 | 15.65 | 0.17 | 15.82 | (0.00) | (-0.11) | (0.00)
1980-1983 | 4.76 | 0.04 | 4.80 | (0.00) | ( 0.00) | (0.00)
1984-1990 | 5.16 | 0.03 | 5.19 | (0.00) | (-0.25) | (0.00)
(c) Transatlantic (encouraging small firms)

Regime | CS | PS | SW | CS change | PS change | SW change
---|---|---|---|---|---|---
1973-1979 | 15.54 | 0.22 | 15.76 | (-0.01) | (0.16) | ( 0.00)
1980-1983 | 4.67 | 0.05 | 4.72 | (-0.02) | (0.25) | (-0.02)
1984-1990 | 5.06 | 0.05 | 5.11 | (-0.02) | (0.25) | (-0.02)
(d) Transatlantic (encouraging large firms)

Regime | CS | PS | SW | CS change | PS change | SW change
---|---|---|---|---|---|---
1973-1979 | 11.80 | 0.16 | 11.96 | (0.00) | (-0.06) | (0.00)
1980-1983 | 3.54 | 0.02 | 3.56 | (0.00) | ( 0.00) | (0.00)
1984-1990 | 3.73 | 0.02 | 3.75 | (0.00) | ( 0.00) | (0.00)
(e) Asia-Europe (encouraging small firms)

Regime | CS | PS | SW | CS change | PS change | SW change
---|---|---|---|---|---|---
1973-1979 | 11.79 | 0.18 | 11.97 | ( 0.00) | (0.06) | ( 0.00)
1980-1983 | 3.53 | 0.03 | 3.56 | ( 0.00) | (0.50) | ( 0.00)
1984-1990 | 3.71 | 0.02 | 3.73 | (-0.01) | (0.00) | (-0.01)
(f) Asia-Europe (encouraging large firms)

Note: Each number is the sum of discounted present values at 1973 for each
regime. Each value is measured by 1 billion USD. The number in a bracket shows
the proportional change from the benchmark model. For example, (-0.50) means
that the hypothetical welfare is $(1-0.50)\times 100=50\%$ of the actual
welfare.

##  7 Conclusion

I investigate how explicit cartels, known as “shipping conferences” in a
global container shipping market facilitated the formation of one of the
largest globally integrated markets through the entry, exit, and shipbuilding
investment of shipping firms. Using a novel data, I develop and construct a
structural model and find that the cartels shifted shipping prices by 20-50%
and encouraged firms’ entry and investment. In my counterfactual simulation, I
compare welfare outcomes with and without the cartels and other inner
allocation rules. I find that cartels significantly increase producer surplus,
slightly decrease consumer surplus, and then may increase social welfare by
encouraging firms’ entry and shipbuilding investment, validating industry
policies controlling prices and quantities in the early stage of the new
industry, which may not be always harmful. Investigating hypothetical
allocation rules supporting large or small firms, I find that the actual rule
based on tonnage shares is the best to maximize social welfare.

A logical next step is to investigate the theoretical and numerical properties
of the equilibrium concept proposed by Igami (2017, 2018). This specific
sequential game focuses on across-level competition, assuming the absence of
within-level competition to mitigate intractability and the potential for
multiple equilibria which is common in Markov perfect equilibria, and aims to
provide a unique equilibrium solution for practical and counterfactual models
through a full-solution solution method. While the utilization of Markov
perfect equilibria involves additional technical procedures for testing and
estimation, as formally discussed in Otsu and Pesendorfer (2023), recent
survey papers on empirical dynamic game literature underscore the importance
of achieving equilibrium uniqueness and tractability. These surveys also
summarize various theoretical conditions (Aguirregabiria et al. 2021).
Alternatively, Abbring and Campbell (2010) develop an entry model with
identical firms, assuming entry and exit decisions follow a last-in-first-out
(LIFO) structure. This model achieves a unique equilibrium by introducing
assumptions about the demand process. It is worth noting that this uniqueness
result typically applies to identical firms. Extending their approach to
nonstationary and asymmetric settings would be a significant contribution to
the literature.

Another avenue for further research is to compare the current model with a
cartel formation and breakdown model that incorporates well-documented
institutional details and data. While this paper primarily focuses on specific
aspects of shipping conferences as a case study, investigating a general
empirical model that encompasses both explicit and implicit cartel models as
summarized in Marshall and Marx (2014) and Asker and Nocke (2021) would be a
valuable contribution to the literature.

## References

  * Abbring and Campbell (2010) Abbring, J. H. and Campbell, J. R. (2010). Last-in first-out oligopoly dynamics. Econometrica, 78 (5), 1491–1527.
  * Aguirregabiria et al. (2021) Aguirregabiria, V., Collard-Wexler, A. and Ryan, S. P. (2021). Dynamic games in empirical industrial organization. In K. Ho, A. Hortaçsu and A. Lizzeri (eds.), Handbook of Industrial Organization, Volume 4, Handbook of Industrial Organization, vol. 4, Elsevier, pp. 225–343.
  * Alé-Chilet and Atal (2020) Alé-Chilet, J. and Atal, J. P. (2020). Trade associations and collusion among many agents: evidence from physicians. The RAND Journal of Economics, 51 (4), 1197–1221.
  * Asker (2010a) Asker, J. (2010a). Leniency and post-cartel market conduct: Preliminary evidence from parcel tanker shipping. International Journal of Industrial Organization, 28 (4), 407–414.
  * Asker (2010b) — (2010b). A study of the internal organization of a bidding cartel. American Economic Review, 100 (3), 724–62.
  * Asker et al. (2019) —, Collard-Wexler, A. and De Loecker, J. (2019). (mis) allocation, market power, and global oil extraction. American Economic Review, 109 (4), 1568–1615.
  * Asker and Nocke (2021) — and Nocke, V. (2021). Collusion, mergers, and related antitrust issues. In K. Ho, A. Hortaçsu and A. Lizzeri (eds.), Handbook of Industrial Organization, Volume 5, Handbook of Industrial Organization, vol. 5, Elsevier, pp. 177–279.
  * Bai and Li (2021) Bai, X. and Li, Y. (2021). The congestion effect of oil transportation and its trade implications. Available at SSRN 3851173.
  * Barwick et al. (2019) Barwick, P. J., Kalouptsidi, M. and Zahur, N. B. (2019). China’s industrial policy: An empirical evaluation. Tech. rep., National Bureau of Economic Research.
  * Bernhofen et al. (2016) Bernhofen, D. M., El-Sahli, Z. and Kneller, R. (2016). Estimating the effects of the container revolution on world trade. Journal of International Economics, 98, 36–50.
  * Brancaccio et al. (2020) Brancaccio, G., Kalouptsidi, M. and Papageorgiou, T. (2020). Geography, transportation, and endogenous trade costs. Econometrica, 88 (2), 657–691.
  * Bresnahan (1982) Bresnahan, T. F. (1982). The oligopoly solution concept is identified. Economics Letters, 10 (1-2), 87–92.
  * Bresnahan (1987) — (1987). Competition and collusion in the american automobile industry: The 1955 price war. The Journal of Industrial Economics, pp. 457–482.
  * Broeze (2002) Broeze, F. (2002). The Globalisation of the Oceans: Containerisation from the 1950s to the Present. 23, Oxford University Press.
  * Byrne and De Roos (2019) Byrne, D. P. and De Roos, N. (2019). Learning to coordinate: A study in retail gasoline. American Economic Review, 109 (2), 591–619.
  * Caoui (2022) Caoui, E. H. (2022). A study of umbrella damages from bid rigging. The Journal of Law and Economics, 65 (2), 239–277.
  * Clark et al. (2018) Clark, R., Coviello, D., Gauthier, J.-F. and Shneyerov, A. (2018). Bid rigging and entry deterrence in public procurement: Evidence from an investigation into collusion and corruption in quebec. The Journal of Law, Economics, and Organization, 34 (3), 301–363.
  * Clark and Houde (2013) — and Houde, J.-F. (2013). Collusion with asymmetric retailers: Evidence from a gasoline price-fixing case. American Economic Journal: Microeconomics, 5 (3), 97–123.
  * Clark and Houde (2014) — and — (2014). The effect of explicit communication on pricing: Evidence from the collapse of a gasoline cartel. The Journal of Industrial Economics, 62 (2), 191–228.
  * Clyde and Reitzes (1998) Clyde, P. S. and Reitzes, J. D. (1998). Market power and collusion in the ocean shipping industry: is a bigger cartel a better cartel? Economic Inquiry, 36 (2), 292–304.
  * Coşar and Demir (2018) Coşar, A. K. and Demir, B. (2018). Shipping inside the box: Containerization and trade. Journal of International Economics, 114, 331–345.
  * Department of Transportation, the United States (1992) Department of Transportation, the United States (1992). Report to the President and the Congress of the Advisory Commission on Conferences in Ocean Shipping. DIANE Publishing.
  * Fan and Xiao (2015) Fan, Y. and Xiao, M. (2015). Competition and subsidies in the deregulated us local telephone industry. the RAND Journal of Economics, 46 (4), 751–776.
  * Fershtman and Pakes (2000) Fershtman, C. and Pakes, A. (2000). A dynamic oligoploy with collusion and price wars. RAND Journal of Economics, 31, 207–236.
  * Fox (1992) Fox, N. R. (1992). An empirical analysis of ocean liner shipping. International Journal of Transport Economics/Rivista internazionale di economia dei trasporti, pp. 205–225.
  * Fusillo (2003) Fusillo, M. (2003). Excess capacity and entry deterrence: the case of ocean liner shipping markets. Maritime Economics & Logistics, 5 (2), 100–115.
  * Gabrielli and Willington (2020) Gabrielli, M. F. and Willington, M. (2020). Assessment of collusion damages in first price auctions. Available at SSRN 3388324.
  * Genesove and Mullin (1998) Genesove, D. and Mullin, W. P. (1998). Testing static oligopoly models: conduct and cost in the sugar industry, 1890-1914. The RAND Journal of Economics, pp. 355–377.
  * Genesove and Mullin (2001) — and — (2001). Rules, communication, and collusion: Narrative evidence from the sugar institute case. American Economic Review, 91 (3), 379–398.
  * Gilpin (1983) Gilpin, K. N. (1983). Hellenic lines in chapter 11. New York Times.
  * Greenwood and Hanson (2015) Greenwood, R. and Hanson, S. G. (2015). Waves in ship prices and investment. The Quarterly Journal of Economics, 130 (1), 55–109.
  * Harrington et al. (2018) Harrington, J. E., Hüschelrath, K. and Laitenberger, U. (2018). Rent sharing to control noncartel supply in the german cement market. Journal of Economics & Management Strategy, 27 (1), 149–166.
  * Igami (2015) Igami, M. (2015). Market power in international commodity trade: The case of coffee. The Journal of Industrial Economics, 63 (2), 225–248.
  * Igami (2017) — (2017). Estimating the innovator’s dilemma: Structural analysis of creative destruction in the hard disk drive industry, 1981–1998. Journal of Political Economy, 125 (3), 798–847.
  * Igami (2018) — (2018). Industry dynamics of offshoring: The case of hard disk drives. American Economic Journal: Microeconomics, 10 (1), 67–101.
  * Igami and Sugaya (2022) — and Sugaya, T. (2022). Measuring the incentive to collude: the vitamin cartels, 1990–99. The Review of Economic Studies, 89 (3), 1460–1494.
  * Igami and Uetake (2020) — and Uetake, K. (2020). Mergers, innovation, and entry-exit dynamics: Consolidation of the hard disk drive industry, 1996–2016. The Review of Economic Studies, 87 (6), 2672–2702.
  * Japan Maritime Center (2008) Japan Maritime Center (2008). Report on Inter-Carrier Agreements in Oceangoing Shipping Industry (Gaikokaiun Ni Okeru Senshakan Kyotei Ni Kakaru Chosa Hokokusho). Accessed: 2022-03-12.
  * Jeon (2022) Jeon, J. (2022). Learning and investment under demand uncertainty in container shipping. The RAND Journal of Economics, 53 (1), 226–259.
  * Juhász et al. (2023) Juhász, R., Lane, N. J. and Rodrik, D. (2023). The New Economics of Industrial Policy. Tech. rep., National Bureau of Economic Research.
  * Kalouptsidi (2014) Kalouptsidi, M. (2014). Time to build and fluctuations in bulk shipping. American Economic Review, 104 (2), 564–608.
  * Kalouptsidi (2017) — (2017). Detection and impact of industrial subsidies: The case of chinese shipbuilding. The Review of Economic Studies, 85 (2), 1111–1158.
  * Kasahara and Shimotsu (2012) Kasahara, H. and Shimotsu, K. (2012). Sequential estimation of structural models with a fixed point constraint. Econometrica, 80 (5), 2303–2319.
  * Krasnokutskaya and Seim (2011) Krasnokutskaya, E. and Seim, K. (2011). Bid preference programs and participation in highway procurement auctions. American Economic Review, 101 (6), 2653–2686.
  * Levenstein and Suslow (2006) Levenstein, M. C. and Suslow, V. Y. (2006). What determines cartel success? Journal of Economic Literature, 44 (1), 43–95.
  * Marin and Sicotte (2003) Marin, P. L. and Sicotte, R. (2003). Exclusive contracts and market power: evidence from ocean shipping. The Journal of Industrial Economics, 51 (2), 193–214.
  * Maritime Industry Research Institute (2004) Maritime Industry Research Institute (2004). Market Structure and Competition Policy in Shipping (Kaiun ni okeru Sijou Kouzou to Kyousou Seisaku). Accessed: 2021-09-22.
  * Marshall and Marx (2014) Marshall, R. C. and Marx, L. M. (2014). The economics of collusion: Cartels and bidding rings. Mit Press.
  * Matsumura and Otani (2023a) Matsumura, Y. and Otani, S. (2023a). Conduct parameter estimation in homogeneous goods markets with equilibrium existence and uniqueness conditions: The case of log-linear specification. Working Paper.
  * Matsumura and Otani (2023b) — and — (2023b). Finite sample performance of a conduct parameter test in homogenous goods markets. arXiv preprint arXiv:2310.04576.
  * Matsumura and Otani (2023c) — and — (2023c). Resolving the conflict on conduct parameter estimation in homogeneous goods markets between bresnahan (1982) and perloff and shen (2012). Economics Letters, p. 111193.
  * Miller and Weinberg (2017) Miller, N. H. and Weinberg, M. C. (2017). Understanding the price effects of the millercoors joint venture. Econometrica, 85 (6), 1763–1791.
  * Morton (1997) Morton, F. S. (1997). Entry and predation: British shipping cartels 1879–1929. Journal of Economics & Management Strategy, 6 (4), 679–724.
  * Otani (2021) Otani, S. (2021). Estimating endogenous coalitional mergers: Merger costs and assortativeness of size and specialization. arXiv preprint arXiv:2108.12744.
  * Otani and Matsuda (2023a) — and Matsuda, T. (2023a). Unified container shipping industry data from 1966: Freight rate, shipping quantity, newbuilding, secondhand, and scrap price. Transportation Research Part E: Logistics and Transportation Review, 176, 103186.
  * Otani and Matsuda (2023b) — and — (2023b). Unified merger list in the container shipping industry from 1966: A structural estimation of the transition of importance of a firm’s age, tonnage capacity, and geographical proximity on merger decision. arXiv preprint arXiv:2310.09938.
  * Otsu and Pesendorfer (2023) Otsu, T. and Pesendorfer, M. (2023). Equilibrium multiplicity in dynamic games: Testing and estimation. The Econometrics Journal, 26 (1), C26–C42.
  * Pace (1981) Pace, E. (1981). Company news; farrell lines profitable again. New York Times.
  * Pirrong (1992) Pirrong, S. C. (1992). An application of core theory to the analysis of ocean shipping markets. The Journal of Law and Economics, 35 (1), 89–131.
  * Podolny and Morton (1999) Podolny, J. M. and Morton, S. F. M. (1999). Social status, entry and predation: The case of british shipping cartels 1879–1929. The Journal of Industrial Economics, 47 (1), 41–67.
  * Porter (1983) Porter, R. H. (1983). A study of cartel stability: The joint executive committee, 1880-1886. The Bell Journal of Economics, pp. 301–314.
  * Röller and Steen (2006) Röller, L.-H. and Steen, F. (2006). On the workings of a cartel: Evidence from the norwegian cement industry. American Economic Review, 96 (1), 321–338.
  * Rua (2014) Rua, G. (2014). Diffusion of containerization. Available at SSRN 2514849.
  * Rust (1987) Rust, J. (1987). Optimal replacement of gmc bus engines: An empirical model of harold zurcher. Econometrica, 55 (5), 999–1033.
  * Ryan (2012) Ryan, S. P. (2012). The costs of environmental regulation in a concentrated industry. Econometrica, 80 (3), 1019–1061.
  * Sjostrom (2013) Sjostrom, W. (2013). Competition and cooperation in liner shipping. In The Handbook of maritime economics and business, Informa Law from Routledge, pp. 463–486.
  * The Transport Policy Council (2007) The Transport Policy Council (2007). Ensuring Stable International Maritime Transport Maritime Policy (Anteiteki Na Kokusai Kaijouyusou Kakuho No Tameno Kaijiseisaku No Arikata Ni Tsuite). Accessed: 2022-03-12.
  * Tiew (2022) Tiew, A. (2022). Flailing Firms and Joint Operating Agreements: An Application to US Local Daily Print Newspapers from 1932 to 1992. Harvard working paper.
  * Van Ham et al. (2012) Van Ham, H., van Ham, J. and Rijsenbrij, J. (2012). Development of containerization: Success through vision, drive and technology. Ios Press.
  * Wilson and Casavant (1991) Wilson, W. W. and Casavant, K. L. (1991). Some market power implications of the shipping act of 1984: a case study of the us to pacific rim transportation markets. Western Journal of Agricultural Economics, pp. 427–434.

Appendix (For Online Publication)

This Appendix is prepared for online publication and contains supplementary
materials. Table 11 lists the contents.

Appendix | Contents |
---|---|---
A | Computational details | Algorithm, Confidence Intervals, Computational time, Discretization
B | Data details | Data construction, Merger issues
C | Model details | The link between my model and Porter (1983)
D | Additional estimation results | Data fitting
Table 11: Contents of the Appendix

##  Appendix A Computation details

###  A.1 Algorithm

I construct the algorithm by applying Igami (2017, 2018) to my model setting.
I fix targeted parameter $\theta_{\pi}$. For exposition, I focus on market
$m=1$. I vectorize all state variables into $s_{mt}$ and it takes a value from
1 to $S$. As in the empirical exercise, I fix the number of potential
entrants, $N_{mt}^{pe}$, to four. Note that, at the equilibrium, fixed points
of CCPs generate corresponding value functions. The algorithm to compute value
functions via backward induction proceeds as follows.

##### Algorithm 1: Value Functions and CCPs

  1. 1.

For $t=T-1,\cdots,1$, obtain the value function at $t+1$ denoted by
$V_{imt+1}(s_{mt+1})$. Given $\theta_{\pi}$, obtain exit cost $\psi$,
operation cost $O$, and investment cost $I$.

    1. (a)

For $s_{mt}=1,\cdots,S$, obtain the number of firms, $N_{mt}$, from $s_{mt}$.

    2. (b)

Initialize and obtain old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$ and old
$Pr(a_{imt}|s_{mt})$ for all $a_{imt}\in\mathcal{A}^{inc}$ and
$i\in\mathcal{N}_{mt}$.

    3. (c)

Initialize and obtain old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$ and old
$Pr(a_{imt}|s_{mt})$ for all $a_{imt}\in\mathcal{A}^{pe}$ and
$i\in\mathcal{N}_{mt}^{pe}$.

    4. (d)

Then, find fixed points of CCPs as follows.

      1. (i)

For level $l=4$ and for all $a_{imt}\in\mathcal{A}^{inc}$ and
$i\in\mathcal{N}_{mt}^{l}$,

        1. A.

Given old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$, old $Pr(a_{imt}|s_{mt})$, $N_{mt}$,
and $N_{mt}^{pe}$, compute the joint likelihood

$\left(P^{pe}(N_{mt}^{pe},X_{mt}^{pe}\mid
s_{mt};\theta_{\pi})\prod_{l=1}^{4}P^{l}(N_{mt}^{l},X_{mt}^{l},B_{mt}^{l}\mid
s_{mt};\theta_{\pi})\right)$

for all $N_{mt}^{pe},X_{mt}^{pe},N_{mt}^{l},X_{mt}^{l},$ and $B_{mt}^{l}$ for
$l=1,2,3,4$.

        2. B.

Given the joint likelihood, $N_{mt}$, and $N_{mt}^{pe}$, compute the
transition probability $P(s_{mt+1}|s_{mt})$ for all $s=1,\cdots,S$.

        3. C.

Given the joint likelihood, the transition probability, and
$V_{imt+1}(s_{mt+1})$, compute the new expectation

$E[V_{imt+1}(s_{mt+1})|s_{mt}]=\sum_{s_{mt+1}=1}^{S}P(s_{mt+1}|s_{mt})V_{imt+1}(s_{mt+1}),$

then update old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$ of corresponding actions to
new $E[V_{imt+1}(s_{mt+1})|s_{mt}]$.

        4. D.

Given the new $E[V_{imt+1}(s_{mt+1})|s_{mt}]$, compute new
$Pr(a_{imt}|s_{mt})$. If the new probability is overflown, go to 1 with new
$\theta_{\pi}$.

        5. E.

Obtain $\text{(CCP gap of
incumbents)}=\text{new}Pr(a_{imt}|s_{mt})-\text{old}Pr(a_{imt}|s_{mt})$.

        6. F.

Update $\text{old}Pr(a_{imt}|s_{mt})$ to
$(\text{new}Pr(a_{imt}|s_{mt})+\text{old}Pr(a_{imt}|s_{mt}))/2$.

        7. G.

Update $l=l-1$ and go to (A) until $l=1$.

      2. (ii)

For potential entrants,

        1. A.

Given the old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$, old $Pr(a_{imt}|s_{mt})$,
$N_{mt}$, and $N_{mt}^{pe}$, compute the joint likelihood.

        2. B.

Given the joint likelihood, $N_{mt}$, and $N_{mt}^{pe}$, compute the
transition probability.

        3. C.

Given the joint likelihood, the transition probability, and
$V_{imt+1}(s_{mt+1})$, compute new $E[V_{imt+1}(s_{mt+1})|s_{mt}]$ from the
old $Pr(a_{imt}|s_{mt})$. Update the old $E[V_{imt+1}(s_{mt+1})|s_{mt}]$ of
corresponding actions to new $E[V_{imt+1}(s_{mt+1})|s_{mt}]$.

        4. D.

Given the new $E[V_{imt+1}(s_{mt+1})|s_{mt}]$, compute new
$Pr(a_{imt}|s_{mt})$. If the new probability is overflown, go to 1 with new
$\theta_{\pi}$.

        5. E.

Obtain $\text{(CCP gap of potential
entrants)}=\text{new}Pr(a_{imt}|s_{mt})-\text{old}Pr(a_{imt}|s_{mt})$.

        6. F.

Update $\text{old}Pr(a_{imt}|s_{mt})$ to
$(\text{new}Pr(a_{imt}|s_{mt})+\text{old}Pr(a_{imt}|s_{mt}))/2$.

      3. (iii)

Check $\text{(CCP gap of incumbents)}+\text{(CCP gap of potential
entrants)}<0.01$. If it is true, $\text{new}Pr(a_{imt}|s_{mt})$ is a fixed
point. Otherwise, go to (i) again.

      4. (iv)

Given $\text{new}Pr(a_{imt}|s_{mt})$ and
$\text{new}E[V_{imt+1}(s_{mt+1})|s_{mt}]$, compute value function
$V_{imt}(s_{mt})$.

By nesting the above Algorithm 1 for value functions and CCPs as an inner
loop, we evaluate the joint log-likelihood $ll_{m}(\theta_{\pi})$ of observing
choices in data as an outer loop. I use the Nelder-Mead algorithm for the
outer loop. For the inner loop 1-(d), I implement policy function iteration
based on the updating rule similar to the algorithm of Kasahara and Shimotsu
(2012) which shows a good contraction property.

###  A.2 Calculation of confidence intervals

I calculate 90% confidence intervals using likelihood ratio tests, which
involve comparing $ll(\hat{\theta}_{\pi})-ll(\tilde{\theta}_{\pi})$ with the
corresponding critical values from a $\chi^{2}$ distribution. Here,
$ll(\theta_{\pi})$ is the joint logarithmic likelihood evaluated at a vector
of parameter values $\hat{\theta}_{\pi}$, and $\tilde{\theta}_{\pi}$
represents the maximum likelihood estimate (MLE) and its perturbed
counterpart, respectively. I do not use standard errors based on Wald-type
tests because the likelihood function exhibits a step function-like shape due
to the game-theoretic nature of the model, as discussed in Igami (2017, 2018)
and Igami and Uetake (2020). Specifically, for each parameter, I calculate the
difference between the original likelihood and the likelihood evaluated at
perturbed parameters, which consist of the four original parameters and one
perturbed parameter. The perturbed parameter is chosen from eleven-grid
intervals centered on the corresponding point-estimated value. Then, find
lower and upper bounds of the perturbed parameter satisfying that
$ll(\hat{\theta}_{\pi})-ll(\tilde{\theta}_{\pi})$ is less than the $\chi^{2}$
critical value for the corresponding degrees of freedom.

###  A.3 Computational time

The main computational burden comes from Algorithm 1, in particular, finding
fixed points for all state variables. Using appropriate starting values, it
takes at least a week for the transatlantic market and two months for the
transpacific market to estimate $\theta_{\pi}$. Depending on the grids of
perturbed parameters, it takes several days for all markets to calculate
confidence intervals. It takes some minutes to simulate 1,000 equilibrium
paths based on estimated CCPs.

###  A.4 Discretization details

To compute equilibrium outcome from discretized tonnage size in the static
part, I need to recover “actual” tonnage size from the corresponding
discretized tonnage size. This procedure depends on the representative value
between grid points. Because I use common grid levels for all three markets,
which are heterogeneous, for consistent comparison in the dynamic part, I
should not use common representative values for all markets. To alleviate this
limit in the view of data-fitting, I specify the representative values for
each market based on fitting the equilibrium prices, quantities, and total
tonnage size to the data. I specify the representative values for the Asia-
Europe market as

$\displaystyle\tilde{s}_{imt}=\begin{cases}10.5\quad&\text{if }s_{imt}=4\\\
10.0\quad&\text{if }s_{imt}=3\\\ 9.0\quad&\text{if }s_{imt}=2\\\
8.0\quad&\text{if }s_{imt}=1.\end{cases}$

For the transpacific market,

$\displaystyle\tilde{s}_{imt}=\begin{cases}10.5\quad&\text{if }s_{imt}=4\\\
9.5\quad&\text{if }s_{imt}=3\\\ 8.5\quad&\text{if }s_{imt}=2\\\
8.0\quad&\text{if }s_{imt}=1.\end{cases}$

For the transatlantic market,

$\displaystyle\tilde{s}_{imt}=\begin{cases}12.1\quad&\text{if }s_{imt}=4\\\
10.1\quad&\text{if }s_{imt}=3\\\ 9.2\quad&\text{if }s_{imt}=2\\\
7.2\quad&\text{if }s_{imt}=1.\end{cases}$

Figure 8 shows that equilibrium elements conditional on the actual states in
the data capture the data patterns well, although some lags and deviations
from the data exist due to the discretization. The equilibrium profits
illustrate the market heterogeneity of the markup across all levels of firms.

(a) Transpacific $P_{mt}$

(b) Transatlantic $P_{mt}$

(c) Asia and Eur $P_{mt}$

(d) Transpacific $Q_{mt}$

(e) Transatlantic $Q_{mt}$

(f) Asia and Eur $Q_{mt}$

(g) Transpacific $S_{mt}$

(h) Transatlantic $S_{mt}$

(i) Asia and Eur $S_{mt}$

(j) Transpacific $\pi_{mt}$

(k) Transatlantic $\pi_{mt}$

(l) Asia and Eur $\pi_{mt}$

Figure 8: Estimated market-year-level prices, quantities, total tonnage, and
firm’s profit of each level.

Note: Each element is derived from the discretized actual state in the data.

##  Appendix B Data details

The route-year-level prices and quantities are constructed based on Otani and
Matsuda (2023a). The industry-year-level merger patterns between 1966 and 2022
are discussed in Otani and Matsuda (2023b). Mergers in the ocean shipping
industry before the introduction of global container transportation are
studied by Otani (2021) regarding Japanese shipping consolidation in 1964. In
the period between 1973 and 1990, two mergers occurred in 1986, three mergers
occurred in 1988, and two mergers occurred in 1990. The final two mergers do
not affect my analysis because the data records ship-level information of
merged firms in 1990. Although the other five mergers may affect my analysis,
I treat merged firms as exiting firms from shipping conferences and merging
firms as investing firms if their state variables in the next period show
level-up due to data limitation because there are at most two mergers at
market-year-level and multiple simultaneous mergers are difficult to model
unlike month-industry-level sequential merger decisions of Igami and Uetake
(2020).

##  Appendix C Model details

###  C.1 The link between my model and Porter (1983)

There is a link between my model and Porter (1983) who studies a “Joint
Executive Committee railroad cartel” in the U.S. railroad industry.
Concretely, my model does not model strategic interaction in the static market
but assumes that the static market is either collusive or perfectly
competitive, and markup is determined by cartel effects which are constant and
exogenously determined by cartels. In contrast, Porter (1983) models strategic
interaction as quantity competition with conduct parameters that determine
perfect collusion to perfect competition, and the markup is determined
endogenously by quantity competition under specific conduct parameters. First,
my model and Porter (1983) are equivalent in competitive markets except for
the tonnage size term which is given in the static model. Second, my model and
Porter (1983) are different in collusive markets because my model is agnostic
about the cartel behavior and treats collusive behavior as a deviation from
perfectly competitive behavior, whereas Porter (1983) assumes that cartels
choose the total quantity to maximize their joint profit. This difference
comes from the difference in how a researcher interprets prices in the
collusive regime in data.

Porter (1983) assumes a constant elasticity demand model,
$\log(Q_{rt})=\alpha_{0}+\alpha_{1}\log(P_{rt})+\zeta_{rt}$ as in (6) where I
use the same demand parameters because it has the same role between my model
and Porter (1983). Porter (1983) also specifies firm $i$’s marginal cost as
$mc_{irt}(q_{irt})=\tilde{\gamma}_{1}(a_{ir}q_{irt}^{\tilde{\gamma}_{1}-1})$
where $a_{ir}$ is a firm-specific shift parameter and $\tilde{\gamma}_{1}(>1)$
is the parameter of the marginal cost different from $\gamma_{1}$ in my model.
Note that since my model assumes
$mc_{irt}(q_{irt},s_{irt})=\gamma_{0}+\gamma_{1}\frac{q_{irt}}{s_{irt}}$ in
(2), the inverse tonnage size term, $1/s_{irt}$, captures the similar but
time-varying firm-specific shift parameter.

The firms’ actions under different behavioral assumptions can then be
summarized by

$\displaystyle
P_{rt}(1+\frac{\theta_{it}}{\alpha_{1}})=mc_{irt}(q_{irt})=a_{ir}\tilde{\gamma}_{1}q_{irt}^{\tilde{\gamma}_{1}-1}$

where $\theta_{it}$ is the conduct parameter Bresnahan (1982). If firms choose
price noncooperatively or at a perfectly competitive level, they price at
marginal cost, and so $\theta_{it}$ equals zero for all $i$ and $t$. If
instead they maximize joint profits, $\theta_{it}$ equals one for all $i$ and
$t$. If firms produce at Cournot output levels, $\theta_{it}$ equals
$q_{irt}/Q_{rt}$, i.e., the firm $i$’s market share in period $t$.

The individual supply equations are weighted by market shares and added up.
Then, we get the industry supply relationship

$\displaystyle
P_{rt}(1+\frac{\theta_{t}}{\alpha_{1}})=\sum_{i\in\mathcal{N}_{rt}}\frac{q_{irt}}{Q_{rt}}mc_{irt}(q_{irt}),$

where
$\theta_{t}=\sum_{i\in\mathcal{N}_{rt}}\frac{q_{irt}}{Q_{rt}}\theta_{it}$.
Porter (1983) shows that firm $i$’s market share, $\frac{q_{irt}}{Q_{rt}}$, is
constant,
$\frac{a_{ir}^{1/(1-\tilde{\gamma}_{1})}}{\sum_{i\in\mathcal{N}_{rt}}a_{ir}^{1/(1-\tilde{\gamma}_{1})}}$
, for all $t$ in each of the three cases above. Note that higher the value of
the firm-specific variable cost shift parameter, $a_{ir}$, lower the market
share of firm $i$. The supply relationship can now be written as

$\displaystyle
P_{rt}(1+\frac{\theta_{t}}{\alpha_{1}})=\tilde{\gamma}_{1}(\sum_{i\in\mathcal{N}_{rt}}a_{i}^{1/(1-\tilde{\gamma}_{1})})^{1-\tilde{\gamma}_{1}}Q_{rt}^{\tilde{\gamma}_{1}-1}.$

Suppose $I_{rt}$ is an indicator variable that equals one when the industry is
in a cooperative regime and equals zero when the industry witnesses a
reversionary episode. Then, the supply relationship of the industry is given
by

$\displaystyle\log P_{rt}$
$\displaystyle=\log\left(\tilde{\gamma}_{1}(\sum_{i\in\mathcal{N}_{rt}}a_{i}^{1/(1-\tilde{\gamma}_{1})})^{1-\tilde{\gamma}_{1}}\right)+(\tilde{\gamma}_{1}-1)\log
Q_{rt}+\log((1+\frac{\theta_{rt}}{\alpha_{1}})^{-1})$
$\displaystyle=\tilde{\beta}_{0}+\tilde{\beta}_{1}\log
Q_{rt}+\log(\frac{\alpha_{1}}{1+\alpha_{1}})I_{rt},$

where
$\tilde{\beta}_{0}=\log\left(\tilde{\gamma}_{1}(\sum_{i\in\mathcal{N}_{rt}}a_{i}^{1/(1-\tilde{\gamma}_{1})})^{1-\tilde{\gamma}_{1}}\right)$
and $\tilde{\beta}_{1}=(\tilde{\gamma}_{1}-1)$. Porter (1983) estimates
$\tilde{\beta}_{0},\tilde{\beta}_{1}$ and
$\log(\frac{\alpha_{1}}{1+\alpha_{1}})$.

If the marginal cost is given by (2) in my model, the supply relationship can
now be written as

$\displaystyle P_{rt}(1+\frac{\theta_{t}}{\alpha_{1}})$
$\displaystyle=\sum_{i\in\mathcal{N}_{rt}}\frac{q_{irt}}{Q_{rt}}(\gamma_{0}+\gamma_{1}\frac{q_{irt}}{s_{irt}})$
$\displaystyle=\sum_{i\in\mathcal{N}_{rt}}\frac{s_{irt}}{S_{rt}}(\gamma_{0}+\gamma_{1}\frac{q_{irt}}{s_{irt}})$
$\displaystyle=\frac{\gamma_{0}}{S_{rt}}\sum_{i\in\mathcal{N}_{rt}}s_{irt}+\frac{\gamma_{1}}{S_{rt}}\sum_{i\in\mathcal{N}_{rt}}q_{irt}$
$\displaystyle=\gamma_{0}+\gamma_{1}\frac{Q_{rt}}{S_{rt}},$

where the second equation comes from the property that tonnage capacity shares
determine service quantity shares, the third equation comes from the constant
cost parameters, and the fourth equation comes from the aggregations of
individual capacities and quantities. First, this relationship implies that my
model and Porter (1983) are equivalent in competitive markets except for the
tonnage size term, $S_{rt}$, which is given in the static model. Second, to
give a structural sense, it is necessary to additionally assume that cartels
choose their total quantity to maximize their partially joint profit, then the
collusive additional markup is $\frac{\theta_{t}P_{rt}}{-\alpha_{1}}$ that
depends on $P_{rt}$. In other words, my model interprets observed collusive
prices as competitive prices plus cartel effects $\tilde{\gamma}$, whereas
Porter (1983) interprets observed collusive prices as
$P_{rt}(1+\frac{\theta_{t}}{\alpha_{1}})$.

In the main text, under the above trade-off, I choose to use the model in
which I am agnostic on cartel behavior. There are some reasons. First,
container shipping firms provide shipping services on their fixed schedules
between ports, so it is difficult to adjust the total service quantity to some
target level. Second, shipping cartels can control prices to compete with non-
conference firms out of the model so a joint profit maximization model based
on quantity competition might be inappropriate. Third, the assumptions of
Porter (1983) that capacity information does not matter in marginal costs and
the output shares of cartel members are relatively stable across the regimes
are inadequate in my data on nonstationary industry history with entry, exit,
and investment of many firms, shown in Section 2. In my model, tonnage
capacity $s_{irt}$ determines the marginal cost efficiency of firm $i$ and
gives a link between static and dynamic parts of investment decisions, that
are ignored by Porter (1983) assuming stationary markets. Fourth, as a
practical reason, the sample size of my route-year-level data is not large
enough for estimating potentially time-varying conduct parameters, discussed
in Matsumura and Otani (2023a, b, c). My modeling choice is similar to Igami
(2015) which recovers marginal costs from competitive regimes and calculates
collusive markups as collusive prices minus the average of recovered marginal
costs without any model of the coffee cartels.

###  C.2 Qualitative analysis of an individual capacity on revenue, total
cost, and profit

I analyze how dynamic decisions changing tonnage capacities affect static
profits. Note that in the static model, capacities for all firms,
$s_{rt}=(s_{irt},s_{-irt})$, are given. Also, the dynamic decisions on
capacities are discretized, so the following analysis provides an approximated
intuition treating capacities as continuous variables.

In Equation (5), firm $i$ in route $r$ in year $t$ has the profit defined as
follows:

$\displaystyle\pi_{irt}(s_{irt},s_{rt},D_{rt})=\underbrace{P_{rt}^{*}(s_{rt},D_{rt})q_{irt}}_{=\text{Revenue}}-\underbrace{\int_{0}^{q_{irt}}mc_{irt}(q,s_{irt}))dq}_{=\text{Total
cost}},$ $\displaystyle\text{s.t.}\quad
q_{irt}=\begin{cases}Q_{rt}^{*}(s_{rt},D_{rt})\omega_{irt}\quad\quad\text{ if
}t\leq 1983\\\
\frac{P_{rt}^{*}(s_{rt},D_{rt})-\gamma_{0}}{\gamma_{1}}s_{irt}\quad\text{
otherwise },\end{cases}\quad\omega_{irt}\in[0,1],\forall
i,\sum_{i=1}^{N_{rt}}\omega_{irt}=1.$

First, I analyze the first term on revenue. Fixing rivals’ capacity,
$s_{-irt}$, and taking the first-order condition with respect to $s_{irt}$,

$\displaystyle\frac{\partial\text{(Revenue)}}{\partial s_{irt}}$
$\displaystyle=\frac{\partial P_{rt}^{*}(s_{rt},D_{rt})}{\partial
s_{irt}}q_{irt}+P_{rt}^{*}(s_{rt},D_{rt})\frac{\partial q_{irt}}{\partial
s_{irt}}$ $\displaystyle=\begin{cases}\underbrace{\frac{\partial
P_{rt}^{*}(s_{rt},D_{rt})}{\partial
s_{irt}}}_{<0}q_{irt}+P_{rt}^{*}(s_{rt},D_{rt})(\underbrace{\frac{\partial
Q_{rt}^{*}(s_{rt},D_{rt})}{\partial
s_{irt}}}_{>0}\omega_{irt}+Q_{rt}^{*}(s_{rt},D_{rt})\frac{\partial\omega_{irt}}{\partial
s_{irt}})\quad\quad\text{ if }t\leq 1983\\\ \underbrace{\frac{\partial
P_{rt}^{*}(s_{rt},D_{rt})}{\partial
s_{irt}}}_{<0}q_{irt}+P_{rt}^{*}(s_{rt},D_{rt})(\underbrace{\frac{\partial
P_{rt}^{*}(s_{rt},D_{rt})}{\partial
s_{irt}}}_{<0}\frac{s_{irt}}{\gamma_{1}}+\frac{P_{rt}^{*}(s_{rt},D_{rt})-\gamma_{0}}{\gamma_{1}})\quad\quad\text{
otherwise },\end{cases}$

where the signs come not from solving fixed points of equilibrium prices and
quantities in Section 3.2 because these cannot be derived as analytical forms.
Due to dependency on the levels of $q_{irt}$, $P_{rt}^{*}(s_{rt},D_{rt})$,
$s_{irt}$, estimated parameters $\gamma_{0}$ and $\gamma_{1}$, and
specification of $\omega_{irt}$, I emphasize that the signs of remaining terms
are not ex-ante determined. In my empirical application, the terms
$\frac{\partial P_{rt}^{*}(s_{rt},D_{rt})}{\partial s_{irt}}$ and
$\frac{\partial P_{rt}^{*}(s_{rt},D_{rt})}{\partial s_{irt}}$ are small
because the marginal impacts of individual firms’ capacities out of more than
ten firms on the equilibrium prices and quantities are limited. Also, under
the standard allocation rule such that firms with larger capacities provide
more service quantities, it holds that $\frac{\partial\omega_{irt}}{\partial
s_{irt}}>0$. Additionally, my static estimation results confirm that
$\gamma_{1}>0$ and $P_{rt}^{*}(s_{rt},D_{rt})-\gamma_{0}>0$ for all routes,
which means that equilibrium prices are larger than the increasing marginal
costs at zero quantity. Given these results of my specific application,
$\frac{\partial\text{(Revenue)}}{\partial s_{irt}}>0$.

Next, I analyze the second term on the total cost. Fixing rivals’ capacity,
$s_{-irt}$, and taking the first-order condition with respect to $s_{irt}$,

$\displaystyle\frac{\partial\text{(Total cost)}}{\partial s_{irt}}$
$\displaystyle=\frac{\partial}{\partial
s_{irt}}(\gamma_{0}q_{irt}+\gamma_{1}\frac{{q_{irt}}^{2}}{2s_{irt}}+C)$
$\displaystyle=-\gamma_{1}\frac{{q_{irt}}^{2}}{2s_{irt}^{2}}<0.$

Therefore, taking the first-order condition of
$\pi_{irt}(s_{irt},s_{rt},D_{rt})$ with respect to $s_{irt}$,

$\displaystyle\frac{\partial\pi_{irt}(s_{irt},s_{rt},D_{rt})}{\partial
s_{irt}}=\frac{\partial\text{(Revenue)}}{\partial
s_{irt}}-\underbrace{\frac{\partial\text{(Total cost)}}{\partial
s_{irt}}}_{<0},$

and given my specific empirical application,
$\frac{\partial\text{(Revenue)}}{\partial s_{irt}}>0$ so that
$\frac{\partial\pi_{irt}(s_{irt},s_{rt},D_{rt})}{\partial s_{irt}}>0$. Figures
8 (j), (k), and (l) show equilibrium profits conditional on actual states for
each capacity level. This shows the above qualitative static-dynamic link that
increasing capacities in the dynamic part increases static profits.

##  Appendix D Additional estimation results

###  D.1 Data fitting of the main models

Figures 9 (a) and (b) illustrate that the estimated route-year-level prices
and quantities given in the demand part fit the data reasonably well,
replicating the regime changes, albeit in a slightly smoother manner. Table 12
shows that estimated market-year-level prices from the supply side model fit
the prices in data reasonably for each competition regime on average. However,
the estimated standard deviation does not fit the standard deviation in the
data because the sample size is small.

(a) $P_{rt}$: Estimated (solid) and data (dotted)

(b) $Q_{rt}$: Estimated (solid) and data (dotted)

Figure 9: Estimated route-year-level prices and quantities. Table 12: Data fit
for shipping prices for each regime.

Regime | Market | (1) Mean $\hat{P}_{mt}$ | (2) Mean $P_{mt}$ | (1)/(2)
---|---|---|---|---
1973-1979 | Asia and Eur | 3974.854 | 4172.739 | 0.953
| Transatlantic | 2864.721 | 2791.502 | 1.026
| Transpacific | 3920.995 | 3810.528 | 1.029
1980-1983 | Asia and Eur | 2569.928 | 2625.379 | 0.979
| Transatlantic | 1848.499 | 2047.711 | 0.903
| Transpacific | 2979.180 | 2724.519 | 1.093
1984-1990 | Asia and Eur | 1671.175 | 1672.949 | 0.999
| Transatlantic | 1186.535 | 1331.900 | 0.891
| Transpacific | 2444.903 | 2297.765 | 1.064

Note: Market aggregates eastbound and westbound routes. Estimated prices
$\hat{P}_{mt}$ are calculated from the results of Table 4b (b).


# An Annotated Glossary for Data Commons, Data Meshes, and Other Data
Platforms

Robert L. Grossman
University of Chicago
and Open Commons Consortium

(February 26, 2021
Revised 2021–2024
Version 1.07 (April 22, 2024) )

## Abstract

Cloud-based data commons, data meshes, data hubs, and other data platforms are
important ways to manage, analyze and share data to accelerate research and to
support reproducible research. This is an annotated glossary of some of the
more common terms used in articles and discussions about these platforms.

##  1 Introduction

Cloud-based data commons, data meshes, data hubs, and other data platforms are
important ways to manage, analyze and share data to accelerate research and to
support reproducible research. This is an annotated glossary of some of the
more common terms used in articles and discussions about these platforms.
Definitions are from [3], [4], or [6], or as otherwise indicated.

Cloud-based data commons have been around for over a decade [5], and this
glossary reflects a broad community consensus about the terms associated with
data commons. On the other hand, data meshes and data fabric are emerging
architectures (at least when this glossary was last updated in 2024) for
managing data within an organization, and there is not yet a consensus on
their definitions. The definitions here around data meshes, data fabrics, and
associated concepts should be thought of as working definitions that are
likely to change in the future.

##  2 Glossary

Application Programmer Interfaces (API). An An application programming
interface (API) is a type of software interface and provides a way for two or
more computer programs or software components to communicate with each other.
Said another way, APIs are software-based intermediaries that provide a way
for two software components to exchange data.

Authorized environment (aka authorized analysis environment). An authorized
environment is a cloud platform workspace or computing infrastructure that is
approved for the use or analysis of controlled access data [6].

Authorized platform identifier (APID). The SAFE Framework [6] assumes that
cloud platforms have a globally unique identifier (GUID) identifying them
called the authorized platform identifier (APID).

Authorized platform network (APN) and authorized platform network identifier
(APNI). The SAFE Framework [6] assumes that cloud platforms form networks
consisting of two or more cloud platforms called authorized platform networks
(APN) that are identified by a globally unique identifier called an authorized
platform network identifier (APNI).

Authorized user. An authorized user is someone who has applied and been
approved for access to controlled-access data.

Authorized region ID. The SAFE Framework [6] assumes that geographic regions
are identified by a globally unique identifier called an Authorized Region ID
(ARID).

Cloud computing. Cloud computing is a model for enabling ubiquitous,
convenient, on-demand network access to a shared pool of configurable
computing resources (e.g., networks, servers, storage, applications, and
services) that can be rapidly provisioned and released with minimal management
effort or service provider interaction [7].

Public cloud service providers provide on-demand delivery of IT resources
through the Internet or private networks with a pay-as-you-go pricing on an
as-needed basis. Private cloud service providers set up and operate their own
cloud computing infrastructure called private clouds for their own
organizations.

Cloud service provider (CSP). See cloud computing.

Commons. A commons is a natural, cultural or digital resource accessible to
all members of a community, or more broadly of a society. Examples include a
pasture for animals to graze in a village, a dog park for dogs in a city
neighborhood, or natural materials such as air, water for society in general.
These resources are held in common, through a partnership, a not-for-profit,
or other entity, but not owned privately for commercial gain [8].

A particularly important type of commons are digital commons containing data.
A specific type of digital commons is a data commons, which are used by
projects and communities to create open digital resources to accelerate the
rate of discovery and increase the impact and the benefits of the data they
hold. More formally, data commons are software platforms that co-locate: 1)
data, 2) cloud-based computing infrastructure, and 3) commonly used software
applications, tools and services to create a resource for managing, analyzing,
integrating and sharing data with a community.

Common Services Operations Center (CSOC). A Common Services Operations Center
is an operations center operated by a commons services provider for setting
up, configuring, operating, and monitoring data commons, data meshes, data
hubs, and other data platforms for managing, analyzing, and sharing data.

Commons Services Provider. A Common Services Provider is an organization that
provides services for setting up, configuring, operating, and monitoring data
commons, data meshes, data hubs, and other data platforms for managing,
analyzing, and sharing data.

Container. A standard unit of software that packages up code and all its
dependencies so the application runs quickly and reliably from one computing
environment to another (for example, Docker).

Data cleaning. By data cleaning, we mean identifying and correcting problems
with data formats, allowed data values, required documentation, etc. of
submitted data.

Data clouds. A data cloud is a cloud computing platform for managing,
analyzing and sharing datasets.

Data commons. A data commons co-locates data with cloud computing
infrastructure and commonly used software services, tools, and applications
for managing, integrating, analyzing and sharing data that are exposed through
web portals and APIs to create an interoperable resource for a research
community. A data commons provides services so that the data is findable,
accessible, interoperable and reusable (FAIR) [12].

Data commons operating models. A standard operating model for a platform is to
separate the platform owner, the platform developer and the platform operator
[10]. This model can be adapted for data commons and related data platforms by
introducing some additional roles, such as: data platform sponsor / owner;
cloud service provider; commons services provider; data platform operator;
and, data platform developer. In this way, the platform operator role is split
between the operator of the cloud services (such as AWS, GCP or Azure); the
operator of the commons services; and, the operator of the data platform
itself, which relies on the cloud services provider and commons services
provider.

Data curation. We define data curation to be the process of identifying a data
model for a study and aligning submitted data to the agreed upon data model.

Data ecosystems aka data meshes. See data meshes. Data meshes in the context
of this article contain multiple data commons, data repositories, cloud
computing infrastructure, cloud based resources, including databases and
knowledgebases, and other applications that can interoperate using a common
set of services (sometimes called mesh services or framework services) for
authentication, authorization, accessing data, analyzing data and related
functions.

Data fabric. Data meshes and data fabrics are emerging architectures for
managing data within an organization. The goal of both architectures is to
improve data management, but they do so through different approaches and with
distinct focuses. A data mesh emphasizes decentralization and domain-specific
ownership of data by subject matter experts. A data fabric focuses on
providing a unified, consistent and integrated approach to data throughout an
organization. A data mesh is a bottom up approach and a data fabric is a top
down approach to data management.

Data hub. A data hub is a data platform in a data mesh that supports search
and discovery of data in 1, 2 or more data platforms in a data mesh.
Typically, the data itself remains in the data repository, data commons, or
other data platform in the mesh, but the metadata for the data is typically
replicated in the data hub. Data hubs often are connected to workspaces or
analysis environments operated by the data hub so the data can be analyzed.

Data lake. A data lake is a system for storing data as objects, where the
objects have an associated GUID and (object) metadata, but there is no data
model for interpreting the data within the object.

Data governance. Data governance is the set of roles, processes, policies,
standards and tools that promote the availability, quality, security and
privacy of an organization’s data. In the context of data platforms for
sharing research data, data governance includes the agreements and processes
for contributing data to a data platform, the agreements and processes for
approving users to access and analyze data on the data platform, the
agreements and processes for redistributing data to other platforms for
analysis, and related activities.

Data harmonization. 1) Data harmonization as the process that brings together
data from multiples sources and applies uniform and consistent processes, such
as uniform quality control metrics to the accepted data; mapping the data to a
common data model; processing the data with common bioinformatics pipelines;
and post-posting the data using common quality control metrics.

2) More precisely, you can separate data harmonization into two phases. Data
harmonization of submitted data is the process of processing data from
different sources and studies and curating and aligning the data to a common
data model. Data harmonization of derived data is the result of processing
harmonized data using a common, uniform set of data processing pipelines
versus each site separately processed data with their own set of pipelines
that may have minor or major differences.

Data meshes (aka data ecosystems). In the context of this article, data meshes
contain multiple data commons, data repositories, cloud computing
infrastructure, cloud based resources, including databases and knowledgebases,
and other applications that can interoperate using a common set of services
(sometimes called mesh services or framework services) for authentication,
authorization, accessing data, analyzing data and related functions.

More generaly, both data meshes and data fabrics are emerging architectures
for managing data within an organization. Quoting from the glossary entry for
data fabrics: The goal of both architectures is to improve data management,
but they do so through different approaches and with distinct focuses. A data
mesh emphasizes decentralization and domain-specific ownership of data by
subject matter experts. A data fabric focuses on providing a unified,
consistent and integrated approach to data throughout an organization. A data
mesh is a bottom up approach and a data fabric is a top down approach to data
management.

Data mesh services (aka data framework services). A core set of software
services (an example of microservices) or authentication, authorization, data
access, metadata access and other services necessary to support and
interoperate data commons, data meshes, data hubs, and workspaces. In an
influential 1984 paper called “End-to-end arguments in systems design [9],”
Jerome Saltzer, David Reed and David Clark argued for using as few software
services as possible when building large complex distributed systems. In the
2018 publication [2], Grossman proposed a similar approach called the narrow
middle architecture for data platforms supporting scientific and biomedical
data. Data mesh services adhere to this principle.

Data object. In cloud computing, a data object consists of data, a key, and
associated metadata. The data can be retrieved using the key and the metadata
associated with a specific data object can be retrieved, but more general
queries are not supported. Amazon’s S3 storage system is a widely used storage
system for data objects. Compare to structured data.

Data platform. A data platform is a type of software platform [4] that brings
together: i) data contributors that provide data to the platform; ii) data
consumers that access and use data from the platform; iii) a platform owner
that is responsible for funding the platform and the data and platform
governance; iv) a platform developer that develops the platform; and a v)
platform operator that operates the platform.

Digital Object Identifier (DOI). A Digital Object Identifier is an identifier
used to permanently and stably identify (usually digital) objects. DOIs
provide a standard mechanism for retrieval of metadata about the object, and
generally a means to access the data object itself. A DOI is a specific ISO
standard for a particular class of persistent identifiers and is supported by
the DOI Foundation.

Data portal. A data portal is a website that provides interactive access to
data in an underlying data management systems, such as a database. Data
commons, data lakes can also have data portals.

FAIR data. FAIR data are data which meet the principles of findability,
accessibility, interoperability, and reusability [12]. There is now an
extensive literature on FAIR data .

Framework services. Framework Services or Data Commons Framework (DCF)
Services is the term used by Gen3 [1] to refer to data mesh services in the
narrow middle architecture, for data meshes, such as the NCI Cancer Research
Data Commons [11]. These are set of standards-based services with open APIs
for authentication, authorization, creating and accessing FAIR data objects,
and for working with bulk structured data in machine readable, self-contained
format.

Globally Unique Identifier (GUID). A GUID is an essentially unique identifier
that is generated by an algorithm so that no central authority is needed, but
rather different programs running in different locations can generate GUID
with a low probability that they will collide. A common format for a GUID is
the hexadecimal representation of a 128 bit binary number.

Microservices. Microservices are a software architecture that organizes
software into small, independent services that communicate over well defined
APIs. These services can be developed, set up, and scaled independently. A
more traditional architecture is to put all the APIs and other required
functionality into a single application. This is sometimes called a monolithic
architecture. Microservices provide important advantages for large-scale
systems that require scalability and must continue to evolve even as their
code base grows very large, but increases the complexity of operating small-
scale systems.

Narrow Middle Architecture. The narrow architecture is an architecture that
applies the end-to-end argument in systems design principle (see the glossary
entry) to data commons and data meshes [2]. With this approach, the fewest
possible services are used for the core of the system (the narrow middle) and
only these services are standardized. With this architecture, the importing,
curation and integration services for getting data into the commons (one
“end”) and the data exploration, analysis and collaboration services for
getting knowledge out of the commons (the other “end”) are not standardized,
but instead are the focus of different competing and innovative efforts until
the community begins to recognize and adopt approaches that seem to be most
effective.

Persistent Identifier (PID). A persistent identifier (PI or PID) is a long-
lasting reference to a document, file, web page, or other digital object. Most
PIDs have a unique identifier which is linked to the current address of the
digital object or metadata associated with the digital..

A Digital Object Identifier (DOI) is a type of a persistent identifier that is
an implementation of the handle architecture and is standardized by the
International Organization for Standardization (ISO). You can turn any DOI
starting with 10 into a URL by adding https://doi.org/ before the DOI as in
https://doi.org/10.1038/s41597-023-02029-x

A Data GUID is a type of persistent identifier designed to support data
objects and standardized by the GA4GH and the Open Commons Consortium.

Platform (aka software platform). Also, see data platform. A platform is an
ecosystem that has four main types of participants [10]. Consumers that use
the product or service provided by the platform. Producers that provide the
product or service. For example, with a data platform (see above), data
contributors are producers and data users (i.e. researchers) are the
consumers. The platform operator operates the platform. The platform sponsor
(aka platform owner) that is responsible for funding and governing the
platform (both data and platform governance), and managing the associated IP.

Platform Governance. Platform governance includes approving cloud platforms as
having the right to distribute datasets to other platforms and to authorized
users on the platforms and approving cloud platforms as authorized
environments so that the cloud platforms can be used by authorized users to
access, analyze, and explore datasets

Platform operator. A platform operator is responsible for operating cloud
platforms that provide access to datasets and to analysis tools and
implementing controls that balance the availability of data with the
obligation to protect the confidentiality and security of the data.

(1) There is an agreement between the Project Sponsor and a Platform Operator
that sets the terms and conditions that enable the cloud platform to
distribute data to authorized users and to transfer the data to authorized
platforms (Right to Distribute Data Agreement). (2) There is an agreement
between the Project Sponsor and a Platform Operator that sets the terms and
conditions that enable the cloud platform to provide an environment to users
so they can access and analyze data (Authorized Environment Agreement).

Structured data. Data is structured if it is organized into records and
fields, with each record consisting of one or more data elements (data
fields). In biomedical data, data fields are often restricted to controlled
vocabularies to make querying them easier.

Workspace. A workspace is another name for a cloud-based environment for
analyzing data (aka an authorized analysis environment).

## References

  * [1] Center for Translational Data Science, University of Chicago.  Welcome to Gen3.  https://gen3.org.
  * [2] Robert Grossman.  A Proposed End-To-End Principle for Data Commons.  Medium, July 6, 2018.  Retrieved from https://medium.com/@rgrossman1/a-proposed-end-to-end-principle-for-data-commons-5872f2fa8a47
  * [3] Robert L. Grossman.  Progress Toward Cancer Data Ecosystems.  The Cancer Journal, 24(3):126–130, May 2018.
  * [4] Robert L. Grossman.  Data Lakes, Clouds, and Commons: A Review of Platforms for Analyzing and Sharing Genomic Data.  Trends in Genetics, 35(3):223–234, March 2019.
  * [5] Robert L Grossman.  Ten lessons for data sharing with a data commons.  Scientific Data, 10(1):120, 2023.
  * [6] Robert L. Grossman, Rebecca R. Boyles, Brandi N. Davis-Dusenbery, Amanda Haddock, Allison P. Heath, Brian D. O’Connor, Adam C. Resnick, Deanne M. Taylor, and Stan Ahalt.  A Framework for the Interoperability of Cloud Platforms: Towards FAIR Data in SAFE Environments.  Scientific Data, 11(1):241, February 2024.
  * [7] P M Mell and T Grance.  The NIST definition of cloud computing.  Technical Report NIST SP 800-145, National Institute of Standards and Technology, Gaithersburg, MD, 2011.  Edition: 0.
  * [8] Elinor Ostrom.  Governing the commons: The evolution of institutions for collective action.  Cambridge university press, 1990.
  * [9] Jerome H Saltzer, David P Reed, and David D Clark.  End-to-end arguments in system design.  ACM Transactions on Computer Systems (TOCS), 2(4):277–288, 1984.
  * [10] Marshall W. Van Alstyne, Geoffrey G. Parker, and Sangeet Paul Choudary.  Pipelines, platforms, and the new rules of strategy.  Harvard business review, 94(4):54–62, 2016.
  * [11] Zhining Wang, Tanja M Davidsen, Gina R Kuffel, KanakaDurga Addepalli, Amanda Bell, Esmeralda Casas-Silva, Hayley Dingerdissen, Keyvan Farahani, Andrey Fedorov, Sharon Gaheen, et al.  Nci cancer research data commons: Resources to share key cancer data.  Cancer Research, 2024.
  * [12] Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J.G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A.C ’t Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons.  The FAIR Guiding Principles for scientific data management and stewardship.  Scientific Data, 3(1):160018, December 2016.