2-10-2 | 88.93 (3.43) | 0.02 (0.02) | 3232.61 (821.74) | 89.8 (3.32) | 0.01 (0.01) | 2885.75 (1018.32) | 91.25 (0.15) | 0.01 (0.01) | 3080.71 (1161.3) | 90.14 (1.89) | 0.02 (0.02) | 3268.12 (587.45) | 91.26 (0.15) | 0.02 (0.02) | 2193.95 (1369.05)
500 | B-PT | 1 | 49.8 (0.61) | 0 (0) | 20.42 (1.98) | 49.93 (0.33) | 0 (0) | 23.71 (4.34) | 50.5 (0.51) | 0 (0) | 24.99 (2.45) | 50.2 (0.61) | 0 (0) | 21.33 (2.91) | 50.2 (0.61) | 0 (0) | 17.34 (2.33)
2 | 52.03 (3.89) | 0 (0) | 718.07 (166.84) | 53.89 (4.94) | 0 (0) | 821.48 (52.94) | 54.96 (5.17) | 0 (0) | 753.47 (277.19) | 49.79 (0.51) | 0 (0) | 632.35 (67.21) | 49.89 (1.17) | 0 (0) | 614.77 (137)
CF | 100 | 90.95 (0.45) | - | 0.69 (0.01) | 91.21 (0.4) | - | 0.68 (0.02) | 91.17 (0.3) | - | 0.67 (0.01) | 90.9 (0.85) | - | 0.67 (0.02) | 89.96 (1.56) | - | 0.69 (0.01)
J-PT | 1 | 60.12 (0.64) | 0 (0) | 1.02 (0.17) | 60.3 (0.69) | 0 (0) | 0.94 (0.04) | 60.12 (0.64) | 0 (0) | 0.95 (0.03) | 60.35 (0.64) | 0 (0) | 0.95 (0.03) | 60.35 (0.64) | 0 (0) | 1.41 (0.48)
2 | 70.36 (0.64) | 0 (0) | 7.49 (2.13) | 70.02 (0.62) | 0 (0) | 5.39 (2.57) | 70.36 (0.64) | 0 (0) | 4.1 (1.18) | 69.91 (0.74) | 0 (0) | 5.91 (3.57) | 69.91 (0.74) | 0 (0) | 7.07 (3.76)
K-PT | 1 | 58.2 (2.72) | 0 (0) | 5.81 (0.75) | 58.24 (2.73) | 0 (0) | 5.83 (0.55) | 59.89 (0.59) | 0 (0) | 5.15 (1.04) | 50.85 (5.97) | 0 (0) | 5.58 (1.3) | 48.22 (2.31) | 0 (0) | 5.17 (1.04)
2 | 61.83 (6.67) | 0 (0) | 138.4 (31.34) | 67.94 (2.83) | 0 (0) | 138.91 (27.41) | 69.57 (1.38) | 0 (0) | 149.05 (24.19) | 50.3 (8.92) | 0 (0) | 135.3 (11.87) | 48.5 (2.24) | 0 (0) | 144.82 (28.23)
PNN | 2-3-2 | 90.79 (0.58) | 0.05 (0.04) | 2996.29 (1350.27) | 91.21 (0.1) | 0.02 (0.03) | 2721.05 (1120.22) | 91.21 (0.08) | 0.02 (0.02) | 2764.36 (1144.47) | 91.28 (0.15) | 0.05 (0.04) | 3514.17 (192.23) | 91.21 (0.13) | 0.11 (0.04) | 3600.13 (0.1)
2-10-2 | 91 (0.34) | 0.06 (0.01) | 3600.15 (0.07) | 91.21 (0.09) | 0.06 (0.01) | 3601.7 (3.09) | 91.2 (0.08) | 0.06 (0.01) | 3600.47 (0.27) | 90.96 (0.69) | 0.06 (0.01) | 3600.93 (1.66) | 91.2 (0.12) | 0.08 (0.01) | 3600.6 (0.48)

Table 2: Complete Results of Simulated Experiments with Athey Model 1 (with
One-Hot Binarization).

Train Data Length | Model | Architec -ture | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | B-PT | 1 | 48.35 (21) | 0 (0) | 17.6 (4.21) | 56.84 (24.71) | 0 (0) | 15.5 (3.94) | 65.29 (28.6) | 0 (0) | 15.84 (1.91) | 82.06 (3.67) | 0 (0) | 16.53 (3.63) | 80.62 (9.47) | 0 (0) | 17.25 (3.96)
2 | 66.67 (16.28) | 0.9 (0.19) | 3600.05 (0.03) | 55.06 (23.48) | 1.38 (0.56) | 3600.05 (0.02) | 71.92 (23.12) | 1.58 (0.21) | 3600.04 (0.02) | 71.69 (3.79) | 1.55 (0.32) | 3600.02 (0.01) | 48.82 (11.58) | 1.28 (0.36) | 3600.03 (0.01)
CF | 100 | 85.05 (0.55) | - | 0.8 (0.03) | 85.05 (0.54) | - | 0.77 (0.02) | 85.05 (0.54) | - | 0.71 (0.01) | 85.05 (0.54) | - | 0.72 (0.01) | 85.05 (0.54) | - | 0.78 (0.02)
J-PT | 1 | 84.25 (2.09) | 0 (0) | 0.28 (0.02) | 84.09 (2.18) | 0 (0) | 0.34 (0.08) | 84.09 (2.18) | 0 (0) | 0.34 (0.08) | 85.05 (0.54) | 0 (0) | 0.35 (0.06) | 84.25 (2.09) | 0 (0) | 0.41 (0.06)
2 | 80.13 (3.2) | 0 (0) | 75.39 (32.19) | 80.39 (3.37) | 0 (0) | 105.34 (24.29) | 82.17 (2.4) | 0 (0) | 83.48 (16.31) | 81.66 (2.24) | 0 (0) | 95.21 (35.06) | 81.9 (1.78) | 0 (0) | 76.47 (28.1)
K-PT | 1 | 54.3 (22.95) | 0 (0) | 3.03 (1.15) | 58 (28.14) | 0 (0) | 2.4 (0.49) | 68.32 (22.86) | 0 (0) | 2.39 (0.63) | 79.79 (2.96) | 0 (0) | 2.8 (0.59) | 65.58 (18.83) | 0 (0) | 2.77 (0.95)
2 | 71.21 (14.69) | 0 (0) | 689.69 (285.66) | 58.57 (20.66) | 0 (0) | 1415.73 (537.86) | 66.52 (15.66) | 0 (0) | 1913.22 (247.75) | 70.15 (5.69) | 0 (0) | 1635.12 (436.59) | 51.22 (4.96) | 0 (0) | 849.38 (320.42)
PNN | 10-3-2 | 85.05 (0.54) | 0 (0.01) | 888.36 (1517.13) | 85.05 (0.54) | 0.01 (0.02) | 1092.54 (1438.89) | 85.05 (0.54) | 0 (0) | 725.32 (1607.54) | 85.05 (0.54) | 0 (0) | 246.13 (313.25) | 85.05 (0.54) | 0 (0.01) | 853.77 (1546.05)
10-10-2 | 85.05 (0.54) | 0.01 (0.01) | 3600.1 (0.05) | 85.05 (0.54) | 0.02 (0.01) | 3600.09 (0.09) | 85.05 (0.54) | 0.01 (0.01) | 3085.44 (1150.71) | 85.05 (0.54) | 0.01 (0.01) | 2483.05 (1580.02) | 85.05 (0.54) | 0.01 (0.01) | 3600.34 (0.61)
500 | B-PT | 1 | 57.47 (7.74) | 0 (0) | 243.19 (21.18) | 73.09 (16.05) | 0 (0) | 247.53 (48.23) | 84.97 (0.4) | 0 (0) | 218.83 (36.82) | 84.97 (0.4) | 0 (0) | 204.44 (14.8) | 84.97 (0.4) | 0 (0) | 238.79 (28.57)
2 | 58.15 (36.55) | 8.64 (5.4) | 3600.02 (0.01) | 61.38 (33.29) | 7.58 (3.57) | 3600.02 (0.01) | 54.11 (36.02) | 9.43 (7.24) | 3600.02 (0.01) | 74.23 (20.98) | 8.39 (5.09) | 3600.01 (0) | 71.75 (25.99) | 7.07 (4.15) | 3600.07 (0.08)
CF | 100 | 84.97 (0.4) | - | 0.76 (0.03) | 84.99 (0.4) | - | 0.73 (0) | 84.99 (0.38) | - | 0.72 (0.01) | 84.97 (0.4) | - | 0.73 (0.02) | 84.97 (0.4) | - | 0.74 (0.01)
J-PT | 1 | 84.97 (0.4) | 0 (0) | 3.5 (0.63) | 84.97 (0.4) | 0 (0) | 2.82 (0.17) | 84.97 (0.4) | 0 (0) | 3.65 (0.92) | 84.97 (0.4) | 0 (0) | 3.13 (0.21) | 84.97 (0.4) | 0 (0) | 3.64 (0.8)
2 | 85 (0.17) | 0 (0) | 1088.41 (533.15) | 85.08 (0.34) | 0 (0) | 1221.06 (217.82) | 85.12 (0.37) | 0 (0) | 1730.08 (649.18) | 84.66 (0.71) | 0 (0) | 1242.12 (388.31) | 85.07 (0.4) | 0 (0) | 882.05 (347.48)
K-PT | 1 | 57.47 (7.74) | 0 (0) | 38.88 (2.61) | 62.21 (14.59) | 0 (0) | 33.48 (2.7) | 76.08 (11.64) | 0 (0) | 40.15 (10.33) | 84.97 (0.4) | 0 (0) | 35.37 (2.86) | 84.97 (0.4) | 0 (0) | 30.43 (4.12)
2 | 73.51 (4.24) | 0.4 (0.08) | 3600.5 (0.11) | 77.08 (7.47) | 0.43 (0.07) | 3600.48 (0.2) | 81.76 (1.98) | 0.4 (0.1) | 3600.68 (0.42) | 78.37 (1.39) | 0.47 (0.09) | 3600.39 (0.19) | 80.25 (4.67) | 0.48 (0.1) | 3600.32 (0.06)
PNN | 10-3-2 | 84.97 (0.4) | 0.04 (0.01) | 3600.09 (0.03) | 84.97 (0.4) | 0.02 (0.01) | 3600.24 (0.23) | 84.97 (0.4) | 0 (0) | 2197.94 (1650.17) | 84.97 (0.4) | 0.02 (0) | 3600.12 (0.03) | 84.97 (0.4) | 0.04 (0.01) | 3600.2 (0.17)
10-10-2 | 84.97 (0.4) | 0.03 (0.01) | 3600.7 (0.42) | 84.97 (0.4) | 0.02 (0.01) | 3605.46 (11.01) | 84.97 (0.4) | 0.01 (0) | 3604.76 (9.72) | 84.97 (0.4) | 0.02 (0) | 3601.56 (1.65) | 84.97 (0.4) | 0.03 (0.01) | 3605.48 (11.47)

Table 3: Complete Results of Simulated Experiments with Athey Model 2 (with
Adapted Binarization).

Train Data Length | Model | Architec -ture | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | B-PT | 1 | 70.28 (24.1) | 0 (0) | 31.27 (3.51) | 92.16 (8.23) | 0 (0) | 44.32 (5.82) | 95.79 (0.17) | 0 (0) | 43.6 (11.36) | 93.92 (4.22) | 0 (0) | 34.86 (9.49) | 71.56 (21.71) | 0 (0) | 32.34 (6.59)
2 | 66.21 (18.77) | 1.59 (0.62) | 3600.06 (0.01) | 89.28 (10.94) | 2.77 (1.65) | 3600.07 (0.04) | 82.13 (9.6) | 4.18 (2.31) | 3600.12 (0.19) | 73.58 (27.11) | 3.68 (1.99) | 3600.07 (0.09) | 45.24 (12.25) | 1.82 (0.93) | 3600.1 (0.06)
CF | 100 | 95.79 (0.17) | - | 0.88 (0.04) | 95.79 (0.17) | - | 0.99 (0.08) | 95.79 (0.17) | - | 1.09 (0.12) | 95.79 (0.17) | - | 0.92 (0.04) | 95.79 (0.17) | - | 0.85 (0.05)
J-PT | 1 | 95.79 (0.17) | 0 (0) | 0.49 (0.08) | 95.79 (0.17) | 0 (0) | 0.41 (0.03) | 95.79 (0.17) | 0 (0) | 0.39 (0.02) | 86.44 (11.51) | 0 (0) | 0.4 (0.01) | 90.34 (5.06) | 0 (0) | 0.42 (0.05)
2 | 88.69 (3.17) | 0 (0) | 158.63 (66.87) | 89.75 (2.77) | 0 (0) | 110.64 (64.14) | 91.1 (4.14) | 0 (0) | 97 (55.53) | 84.83 (5.74) | 0 (0) | 126.96 (54.38) | 82.29 (8.02) | 0 (0) | 123.41 (42.14)
K-PT | 1 | 70.28 (24.1) | 0 (0) | 5.84 (1.02) | 82.85 (18.07) | 0 (0) | 6 (1.72) | 82.85 (4.96) | 0 (0) | 6.16 (1.61) | 86.8 (0.5) | 0 (0) | 7.11 (0.62) | 67.49 (17.58) | 0 (0) | 4.31 (1.18)
2 | 79.68 (14.99) | 0.02 (0.05) | 2821.78 (738.69) | 87.85 (13.01) | 0.18 (0.11) | 3383.59 (484.22) | 81.17 (5.06) | 0.27 (0.08) | 3600.23 (0.11) | 65.29 (14.41) | 0.13 (0.12) | 3244.13 (682.43) | 45.26 (12.04) | 0 (0) | 1356.21 (771.37)
PNN | 20-3-2 | 95.79 (0.17) | 0 (0) | 0.53 (0.04) | 95.79 (0.17) | 0 (0) | 0.59 (0.08) | 95.79 (0.17) | 0 (0) | 0.54 (0.06) | 95.79 (0.17) | 0 (0) | 0.68 (0.31) | 95.79 (0.17) | 0 (0) | 0.72 (0.31)
20-10-2 | 95.79 (0.17) | 0 (0) | 3.35 (1.47) | 95.79 (0.17) | 0 (0) | 3.57 (1.24) | 95.79 (0.17) | 0 (0) | 3.57 (1.89) | 95.79 (0.17) | 0 (0) | 5.74 (3.16) | 95.79 (0.17) | 0 (0) | 4.61 (2.31)
500 | B-PT | 1 | 78.92 (23.01) | 0 (0) | 660.69 (255.35) | 95.71 (0.15) | 0 (0) | 722.24 (236.38) | 95.71 (0.15) | 0 (0) | 869.83 (285.7) | 95.71 (0.15) | 0 (0) | 541.49 (132.78) | 95.71 (0.15) | 0 (0) | 695.11 (253.31)
2 | 95.71 (0.15) | inf | 3600.06 (0.02) | 95.71 (0.15) | inf | 3600.08 (0.04) | 77.38 (40.93) | inf | 3600.04 (0.04) | 40.74 (50.05) | inf | 3600.02 (0.02) | 28.13 (38.46) | inf | 3600.03 (0.03)
CF | 100 | 95.71 (0.15) | - | 0.81 (0.03) | 95.71 (0.15) | - | 0.8 (0.02) | 95.71 (0.15) | - | 0.79 (0.01) | 95.71 (0.15) | - | 0.79 (0.02) | 95.71 (0.15) | - | 0.82 (0.02)
J-PT | 1 | 95.71 (0.15) | 0 (0) | 4.71 (0.76) | 95.71 (0.15) | 0 (0) | 4.25 (0.37) | 95.71 (0.15) | 0 (0) | 4.39 (0.57) | 95.71 (0.15) | 0 (0) | 3.97 (0.13) | 95.71 (0.15) | 0 (0) | 3.94 (0.52)
2 | 94.68 (0.4) | 0 (0) | 1505.45 (387.84) | 94.8 (0.26) | 0 (0) | 2741.94 (865.17) | 94.77 (0.26) | 0 (0) | 2781.14 (464.43) | 94.78 (0.23) | 0 (0) | 2685.15 (626.67) | 94.81 (0.25) | 0 (0) | 2081.06 (533.69)
K-PT | 1 | 78.92 (23.01) | 0 (0) | 73.35 (12.37) | 95.71 (0.15) | 0 (0) | 81.45 (7.94) | 95.71 (0.15) | 0 (0) | 73.62 (20.97) | 95.71 (0.15) | 0 (0) | 66.69 (15.77) | 91.98 (5.19) | 0 (0) | 78.72 (25.57)
2 | 80.25 (16.15) | 0.54 (0.07) | 3600.58 (0.17) | 94.53 (2.74) | 0.62 (0.19) | 3600.39 (0.09) | 94.01 (2.27) | 0.62 (0.15) | 3600.37 (0.12) | 91.61 (2.83) | 0.73 (0.13) | 3600.42 (0.16) | 84.54 (2.82) | 0.76 (0.13) | 3601.19 (1.15)
PNN | 20-3-2 | 95.71 (0.15) | 0.01 (0) | 3600.37 (0.5) | 95.71 (0.15) | 0 (0) | 311.09 (280.68) | 95.71 (0.15) | 0 (0) | 23.81 (10.85) | 95.71 (0.15) | 0 (0) | 653.81 (775.29) | 95.71 (0.15) | 0.01 (0.01) | 3600.32 (0.36)
20-10-2 | 95.71 (0.15) | 0.01 (0) | 3600.49 (0.39) | 95.71 (0.15) | 0.01 (0) | 3600.58 (0.53) | 95.71 (0.15) | 0 (0) | 1529.09 (1891.52) | 95.71 (0.15) | 0 (0) | 3603.89 (7.25) | 95.71 (0.15) | 0.01 (0) | 3600.49 (0.5)

Table 4: Complete Results of Simulated Experiments with Athey Model 3 (with
Adapted Binarization).

####  D.1.2 Personalized Postpartum Hypertension Treatments

Here we display the average average optimality gap, average running time in
seconds, and the average $\pi$ value for the polices derived by the models in
the postpartum hypertension experiments. The averages are taken over the 10
cross-validation folds in the experiments, with the standard deviation given
parenthetically. The 90% and 95% confidence intervals for the $\pi$ value are
also included. The results of all three experiments are given.

Experiment |  Model |  Model Architecture |  Average Gap (%) with st.dev |  Average Runtime (s) with st.dev |  $\pi$ |  95% Confidence Interval |  90% Confidence Interval
---|---|---|---|---|---|---|---
1 | Policy in Data | - | - | - | 148.32 | [146.59, 150.06] | [146.97, 149.67]
B-PT | 1 | 0.0 (0.0) | 21.74 (1.89) | 146.43 | [145.2, 147.66] | [145.47, 147.39]
2 | 74.84 (13.44) | 3600.15 (0.05) | 147.15 | [145.85, 148.44] | [146.14, 148.16]
CF | 100 | - | 0.14 (0.01) | 146.37 | [145.14, 147.59] | [145.41, 147.32]
J-PT | 1 | 0.05 (0.03) | 3.11 (1.33) | 146.48 | [145.25, 147.71] | [145.52, 147.44]
2 | 0.01 (0.02) | 285.77 (28.25) | 148.58 | [147.32, 149.84] | [147.6, 149.56]
K-PT | 1 | 0.0 (0.0) | 3.96 (0.47) | 146.43 | [145.2, 147.66] | [145.47, 147.39]
2 | 2.32 (5.48) | 2620.15 (519.68) | 147.22 | [146.04, 148.4] | [146.3, 148.14]
PNN | 22-3-2 | 0.03 (0.0) | 3600.42 (0.49) | 146.43 | [145.2, 147.66] | [145.47, 147.39]
22-10-2 | 0.0 (0.0) | 2333.51 (1156.53) | 145.76 | [143.71, 147.81] | [144.16, 147.36]
2 | Policy in Data | - | - | - | 154.92 | [144.58, 165.25] | [146.87, 162.97]
B-PT | 1 | 0.02 (0.04) | 33.21 (13.37) | 155.72 | [145.57, 165.86] | [147.81, 163.62]
2 | 40.18 (21.48) | 3600.15 (0.07) | 155.19 | [145.12, 165.27] | [147.34, 163.05]
CF | 100 | - | 0.17 (0.0) | 150.05 | [148.47, 151.64] | [148.82, 151.29]
J-PT | 1 | 0.03 (0.04) | 8.94 (4.02) | 149.15 | [147.19, 151.1] | [147.62, 150.67]
2 | 0.0 (0.01) | 434.55 (41.5) | 150.05 | [148.17, 151.94] | [148.58, 151.52]
K-PT | 1 | 0.0 (0.0) | 5.61 (1.51) | 157.32 | [147.22, 167.42] | [149.45, 165.19]
2 | 0.0 (0.0) | 1507.98 (483.29) | 156.16 | [146.19, 166.13] | [148.39, 163.93]
PNN | 19-3-4 | 0.03 (0.0) | 3600.14 (0.13) | 148.1 | [146.32, 149.88] | [146.71, 149.49]
19-10-4 | 0.01 (0.0) | 3600.97 (1.5) | 147.3 | [144.48, 150.11] | [145.1, 149.49]
3 | Policy in Data | - | - | - | 150.86 | [146.34, 155.38] | [147.34, 154.38]
B-PT | 1 | 0.01 (0.03) | 37.02 (15.84) | 148.52 | [145.98, 151.07] | [146.54, 150.51]
2 | 0.53 (0.9) | 3156.02 (416.63) | 151.99 | [148.74, 155.25] | [149.46, 154.53]
CF | 100 | - | 0.17 (0.01) | 147.39 | [145.05, 149.73] | [145.56, 149.21]
J-PT | 1 | 0.02 (0.03) | 12.77 (5.65) | 146.7 | [144.27, 149.14] | [144.8, 148.6]
2 | 0.01 (0.02) | 665.01 (165.11) | 148.5 | [146.28, 150.72] | [146.77, 150.23]
K-PT | 1 | 0.0 (0.0) | 6.39 (3.89) | 148.32 | [145.78, 150.86] | [146.34, 150.3]
2 | 21.38 (33.41) | 1855.24 (1507.95) | 151.89 | [149.26, 154.51] | [149.84, 153.93]
PNN | 18-3-7 | 0.09 (0.01) | 3600.16 (0.16) | 146.05 | [144.07, 148.03] | [144.5, 147.59]
18-10-7 | 0.09 (0.01) | 3600.34 (0.1) | 145.38 | [143.65, 147.12] | [144.03, 146.74]

Table 5: Complete Results of Postpartum Hypertension Experiments.

####  D.1.3 Personalized Warfarin Dosing

Table 6 displays the average average optimality gap, average running time in
seconds, and the average $\pi$ value for the polices derived by the models in
the warfarin dosing experiments. The averages are taken over the 5 seeds of
the experiments, with the standard deviation given parenthetically. The 90%
and 95% confidence intervals for the $\pi$ value are also included.

Method |  Model Architecture |  Average Gap (%) with st.dev |  Average Runtime (s) with st.dev |  $\pi$ |  95% Confidence Interval |  90% Confidence Interval
---|---|---|---|---|---|---
Policy in Data | - | - | - | 0.609 | [0.325, 0.892] | [0.388, 0.829]
B-PT | 1 | 0.0 (0.0) | 1185.55 (146.99) | 0.510 | [0.271, 0.749] | [0.324, 0.696]
2 | 100.0 (0.0) | 3600.05 (0.01) | 0.545 | [0.312, 0.779] | [0.363, 0.727]
CF | 100 | - | 0.85 (0.02) | 0.265 | [0.248, 0.282] | [0.252, 0.279]
J-PT | 1 | 0.0 (0.0) | 89.52 (32.72) | 0.232 | [0.182, 0.283] | [0.193, 0.272]
2 | 146.35 (46.04) | 3600.38 (0.4) | 0.275 | [0.265, 0.284] | [0.267, 0.282]
K-PT | 1 | 0.0 (0.0) | 211.66 (63.94) | 0.506 | [0.267, 0.744] | [0.32, 0.692]
2 | 100.0 (0.0) | 3600.3 (0.08) | 0.449 | [0.205, 0.693] | [0.259, 0.639]
PNN | 76-3-2 | 73.6 (82.02) | 3170.79 (867.48) | 0.295 | [0.281, 0.309] | [0.284, 0.306]
76-10-2 | 92.78 (47.38) | 3600.79 (0.9) | 0.290 | [0.271, 0.310] | [0.275, 0.305]

Table 6: Complete Results of Warfarin Dosing Experiments.

####  D.1.4 Prediction Problem Results

We provide the complete results of the prediction experiments conducted over
the 2-digit MNIST problem. Table 7 shows the results of training ANNs over
various architectures for SGD with binary and ReLU activations, and the MIP
with binary activations. The architecture is denoted by (Number of Hidden
Layers, Number of Neurons in Each Layer). So, for example, (2,10) indicates 2
hidden layers with 10 neurons in each layer.

Training Approach |  Hidden Layer Activation |  Architecture |  Average Runtime (s) |  Average Optimality Gap (%) |  Aveage Final Training Accuracy (%) |  Final Testing Average Accuracy (%)
---|---|---|---|---|---|---
SGD | Binary | (1, 3) | 165.31 | - | 79.0 | 77.3
(1, 5) | 195.42 | - | 81.0 | 79.3
(1, 10) | 181.83 | - | 80.0 | 80.00
(2, 3) | 197.65 | - | 58.0 | 61.00
(2, 5) | 207.99 | - | 62.0 | 65.3
(2, 10) | 207.49 | - | 79.0 | 77.7
(3, 3) | 223.53 | - | 50.0 | 50.3
(3, 5) | 239.01 | - | 51.0 | 48.9
(3, 10) | 242.44 | - | 61.0 | 67.0
ReLU | (1, 3) | 125.49 | - | 50.0 | 50.3
(1, 5) | 132.43 | - | 50.0 | 50.3
(1, 10) | 138.27 | - | 50.0 | 50.3
(2, 3) | 149.75 | - | 50.0 | 50.3
(2, 5) | 155.35 | - | 50.0 | 50.3
(2, 10) | 159.56 | - | 54.0 | 53.4
(3, 3) | 169.57 | - | 50.0 | 50.3
(3, 5) | 174.85 | - | 50.0 | 50.3
(3, 10) | 181.34 | - | 50.0 | 50.3
MIP | Binary | (1, 3) | 68.88 | 0.00 | 98.4 | 84.8
(1, 5) | 37.63 | 0.00 | 99.2 | 84.8
(1, 10) | 34.08 | 0.00 | 98.8 | 82.2
(2, 3) | 125.75 | 0.00 | 60.8 | 62.4
(2, 5) | 431.15 | 0.00 | 100.0 | 80.3
(2, 10) | 1056.70 | 0.00 | 99.6 | 87.0
(3, 3) | 238.75 | 0.00 | 100.0 | 90.1
(3, 5) | 135.02 | 0.00 | 99.2 | 85.9
(3, 10) | 3601.15 | 100.00 | 75.6 | 66.2

Table 7: Complete Results of Prediction Experiments.

###  D.2 Hyperparameter Tuning Results for Prescriptive Neural Networks

This sections provides the complete experimental results of the hyperparameter
tuning conducted for the PNN models simulated, postpartum hypertension, and
warfarin dosing experiments. These hyperparameters were tuned with a 10-fold
cross validation set-up, where we considered
$\lambda\in\\{0,0.01,0.1,1,10\\}$. The hyperparameter with the best out-of-
sample probability of correct treatment assignment or $\pi$ was chosen for the
final experiments. The tables are set up in this section are similarly to the
previous section.

####  D.2.1 Simulated Data

Please see Tables 8-13.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 87.6 (2.6) | 0.8 (1) | 2165.3 (1597.7) | 87.1 (1.2) | 2.5 (1.6) | 2881.4 (1607.2) | 87.1 (1.5) | 1.5 (1.4) | 2701.7 (1315.7) | 86.9 (3.2) | 2.8 (1.5) | 3600.4 (0.5) | 86.9 (1.8) | 1.6 (3.5) | 1925.3 (1390.7)
0.01 | 88.9 (3.4) | 0 (0) | 22.6 (14.5) | 89.7 (3.4) | 0 (0) | 16 (10.9) | 91.3 (0.1) | 0 (0) | 11.6 (2.7) | 90.1 (2) | 0 (0) | 14.8 (5.4) | 91.3 (0.2) | 0 (0) | 14.8 (10)
0.1 | 49.6 (0.4) | 0 (0) | 3.2 (0.9) | 49.5 (0.3) | 0 (0) | 2.7 (0.7) | 49.6 (0.4) | 0 (0) | 2.9 (0.8) | 49.5 (0.3) | 0 (0) | 2.9 (0.5) | 50.4 (0.4) | 0 (0) | 3.1 (0.5)
1 | 49.6 (0.4) | 0 (0) | 1.3 (0.3) | 49.5 (0.3) | 0 (0) | 1.1 (0.2) | 49.6 (0.4) | 0 (0) | 1 (0.2) | 49.5 (0.3) | 0 (0) | 0.7 (0.3) | 50.4 (0.4) | 0 (0) | 1.2 (0.2)
10 | 49.7 (0.5) | 0.1 (0) | 0.9 (0.2) | 49.5 (0.3) | 0.1 (0) | 0.9 (0.4) | 49.6 (0.4) | 0.1 (0) | 0.9 (0.1) | 49.5 (0.3) | 0 (0) | 0.8 (0.2) | 50.4 (0.4) | 0.1 (0) | 0.9 (0.1)
10 | 0 | 86.2 (1.8) | 0 (0) | 61.1 (78.8) | 85.4 (1.5) | 0 (0) | 310 (394) | 84.6 (2.3) | 0 (0) | 137.9 (114.5) | 83 (3.9) | 0.1 (0.1) | 930.6 (1518.6) | 86.8 (2.9) | 0 (0) | 134 (114.4)
0.01 | 88.9 (3.4) | 2.4 (2) | 3232.6 (821.7) | 89.8 (3.3) | 1.5 (1.5) | 2885.7 (1018.3) | 91.2 (0.2) | 1.2 (1.1) | 3080.7 (1161.3) | 90.1 (1.9) | 1.8 (1.7) | 3268.1 (587.5) | 91.3 (0.1) | 1.7 (2.3) | 2193.9 (1369)
0.1 | 49.6 (0.4) | 0 (0) | 20.2 (12.4) | 49.5 (0.3) | 0 (0) | 15.7 (4.5) | 49.6 (0.4) | 0 (0) | 15.9 (6.6) | 49.5 (0.3) | 0 (0) | 12.2 (4.2) | 50.4 (0.4) | 0 (0) | 21.2 (3.5)
1 | 49.6 (0.4) | 0.1 (0) | 4.4 (0.6) | 49.5 (0.3) | 0.1 (0) | 4.2 (1.1) | 49.5 (0.3) | 0.1 (0) | 4 (1.2) | 49.6 (0.4) | 0.1 (0) | 4.6 (0.2) | 50 (0.6) | 0.1 (0) | 4 (0.2)
10 | 49.6 (0.4) | 0 (0) | 2.2 (0.1) | 49.5 (0.3) | 0 (0) | 2.2 (0.5) | 49.5 (0.3) | 0 (0) | 2 (0.5) | 49.6 (0.4) | 0 (0) | 2.1 (0.2) | 50 (0.6) | 0 (0) | 2.3 (0.4)
500 | 3 | 0 | 90 (0.9) | 8.9 (1.1) | 3600.3 (0.2) | 90.5 (0.7) | 6.8 (1.3) | 3600.2 (0.1) | 90.3 (0.4) | 5.7 (1.2) | 3600.4 (0.5) | 90.3 (0.4) | 7.3 (2.2) | 3600.1 (0.1) | 89.7 (0.8) | 12.7 (1.9) | 3600.4 (0.5)
0.01 | 90.8 (0.6) | 5.5 (3.7) | 2996.3 (1350.3) | 91.2 (0.1) | 2 (2.8) | 2721.1 (1120.2) | 91.2 (0.1) | 2.2 (2.2) | 2764.4 (1144.5) | 91.3 (0.1) | 5.4 (4) | 3514.2 (192.2) | 91.2 (0.1) | 10.7 (3.5) | 3600.1 (0.1)
0.1 | 50.3 (0.6) | 0 (0) | 67.7 (23.4) | 50.4 (0.5) | 0 (0) | 51.4 (21.9) | 50.1 (0.6) | 0 (0) | 62.1 (7.6) | 50.3 (0.6) | 0 (0) | 51.3 (13.4) | 50.1 (0.6) | 0 (0) | 66.9 (25.7)
1 | 50.3 (0.6) | 0 (0) | 6.5 (1.8) | 50.4 (0.5) | 0 (0) | 6.5 (1.4) | 50.1 (0.6) | 0 (0) | 6.7 (1.3) | 50.3 (0.6) | 0 (0) | 6.7 (0.9) | 50.1 (0.6) | 0 (0) | 5.9 (1.5)
10 | 50.3 (0.6) | 0.1 (0) | 6.8 (1.3) | 50.4 (0.5) | 0 (0) | 6.3 (1.2) | 50.1 (0.6) | 0 (0) | 5.1 (1) | 50.4 (0.5) | 0.1 (0) | 5.2 (0.7) | 50.1 (0.6) | 0 (0) | 5.2 (0.7)
10 | 0 | 89.6 (0.9) | 7 (2.1) | 3600.6 (1.1) | 90 (0.4) | 6.3 (1.1) | 3600.4 (0.2) | 90.3 (0.6) | 5.3 (0.9) | 3601.5 (2.6) | 89.9 (0.6) | 6.8 (2.4) | 3600.3 (0.1) | 88.6 (1.3) | 11.1 (2.8) | 3600.3 (0.3)
0.01 | 91 (0.3) | 6.1 (1) | 3600.1 (0.1) | 91.2 (0.1) | 5.9 (1.3) | 3601.7 (3.1) | 91.2 (0.1) | 5.8 (0.7) | 3600.5 (0.3) | 91 (0.7) | 6.3 (1.2) | 3600.9 (1.7) | 91.2 (0.1) | 7.9 (0.8) | 3600.6 (0.5)
0.1 | 50.3 (0.6) | 0.4 (0.9) | 1424.9 (1445.3) | 50.4 (0.5) | 0.9 (1.2) | 2476.9 (1386.2) | 50.1 (0.6) | 0.8 (1.1) | 2456.3 (1446) | 50.3 (0.6) | 0 (0) | 1372 (1426.7) | 50.1 (0.6) | 0.5 (1) | 2114.2 (1599.4)
1 | 50.3 (0.6) | 0 (0) | 30.3 (8.9) | 50.4 (0.5) | 0 (0) | 33.9 (10.4) | 50.1 (0.6) | 0.1 (0) | 35.4 (2.6) | 50.3 (0.6) | 0.1 (0) | 32.7 (9.1) | 50.1 (0.6) | 0 (0) | 32 (8.3)
10 | 50.3 (0.6) | 0 (0) | 19.9 (4.3) | 50.4 (0.5) | 0 (0) | 20.9 (2.9) | 50.1 (0.6) | 0 (0) | 18.7 (3.3) | 50.1 (0.6) | 0 (0) | 18 (2.9) | 50.2 (0.6) | 0 (0) | 19 (2)

Table 8: PNN Hyperparamter Tuning Results for 2-covariate data (Athey Model 1)
with $\ell_{0}$ Regularization.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 77.9 (4.3) | 0 (0) | 6.1 (5.9) | 76.2 (4.7) | 0 (0) | 14.2 (14.8) | 78.1 (2.6) | 0 (0) | 6.3 (3.2) | 78.3 (3.6) | 0 (0) | 4.1 (1.3) | 79.7 (2) | 0 (0) | 3.7 (3.2)
0.01 | 85.1 (0.5) | 0.4 (0.9) | 888.4 (1517.1) | 85.1 (0.5) | 1.1 (2.4) | 1092.5 (1438.9) | 85.1 (0.5) | 0.1 (0.3) | 725.3 (1607.5) | 85.1 (0.5) | 0 (0) | 246.1 (313.3) | 85.1 (0.5) | 0.4 (0.8) | 853.8 (1546)
0.1 | 85.1 (0.5) | 0 (0) | 2.6 (0.4) | 85.1 (0.5) | 0 (0) | 3.2 (0.9) | 85.1 (0.5) | 0.1 (0) | 2.4 (0.3) | 85.1 (0.5) | 0 (0) | 2.3 (0.4) | 85.1 (0.5) | 0 (0) | 2.6 (0.8)
1 | 85.1 (0.5) | 0.1 (0) | 1.8 (0.7) | 85.1 (0.5) | 0 (0) | 2.3 (0.8) | 85.1 (0.5) | 0.1 (0) | 1.3 (1.1) | 85.1 (0.5) | 0.1 (0) | 1.5 (0.9) | 85.1 (0.5) | 0.1 (0) | 1.6 (0.7)
10 | 85.1 (0.5) | 0 (0) | 1.3 (0.9) | 85.1 (0.5) | 0 (0) | 0.8 (0.2) | 85.1 (0.5) | 0 (0) | 0.9 (0.4) | 85.1 (0.5) | 0 (0) | 0.9 (0.5) | 85.1 (0.5) | 0 (0) | 0.8 (0.1)
10 | 0 | 75.8 (3.4) | 0 (0) | 6.5 (3.3) | 76.6 (2.4) | 0 (0) | 6.5 (3) | 76.5 (2.3) | 0 (0) | 7 (3.1) | 76.3 (1.5) | 0 (0) | 8.9 (8.1) | 79.3 (1.5) | 0 (0) | 4.3 (1.9)
0.01 | 85.1 (0.5) | 1.4 (1) | 3600.1 (0.1) | 85.1 (0.5) | 1.9 (1.3) | 3600.1 (0.1) | 85.1 (0.5) | 0.7 (0.9) | 3085.4 (1150.7) | 85.1 (0.5) | 0.9 (0.9) | 2483 (1580) | 85.1 (0.5) | 1.2 (0.9) | 3600.3 (0.6)
0.1 | 85.1 (0.5) | 0 (0) | 9.5 (0.7) | 85.1 (0.5) | 0 (0) | 11.6 (3.9) | 85.1 (0.5) | 0 (0) | 8.7 (0.8) | 85.1 (0.5) | 0.1 (0) | 8.3 (3) | 85.1 (0.5) | 0.1 (0) | 12.1 (4.6)
1 | 85.1 (0.5) | 0 (0) | 3.8 (1) | 85.1 (0.5) | 0 (0) | 5.5 (2) | 85.1 (0.5) | 0 (0) | 5.2 (1.6) | 85.1 (0.5) | 0 (0) | 4.6 (1.3) | 85.1 (0.5) | 0 (0) | 6.5 (2.3)
10 | 85.1 (0.5) | 0 (0) | 4 (1.2) | 85.1 (0.5) | 0 (0) | 4.7 (2.3) | 85.1 (0.5) | 0 (0) | 4.7 (1.3) | 85.1 (0.5) | 0 (0) | 5.9 (1.4) | 85.1 (0.5) | 0 (0) | 5.3 (1.5)
500 | 3 | 0 | 81.9 (1.5) | 2.5 (1.4) | 3600.2 (0.4) | 82.3 (0.7) | 1.9 (0.4) | 3600.2 (0.1) | 82.4 (0.9) | 1.4 (0.3) | 3600.1 (0.1) | 82.3 (0.8) | 2.7 (0.3) | 3600.1 (0.1) | 81.7 (0.6) | 2.9 (1) | 3600.3 (0.4)
0.01 | 85 (0.4) | 3.5 (1.5) | 3600.1 (0) | 85 (0.4) | 2 (0.9) | 3600.2 (0.2) | 85 (0.4) | 0.2 (0.2) | 2197.9 (1650.2) | 85 (0.4) | 1.9 (0.4) | 3600.1 (0) | 85 (0.4) | 4.3 (1) | 3600.2 (0.2)
0.1 | 85 (0.4) | 0 (0) | 32.7 (28) | 85 (0.4) | 0 (0) | 39.6 (11) | 85 (0.4) | 0.1 (0) | 34.4 (16.9) | 85 (0.4) | 0 (0) | 37.3 (7) | 85 (0.4) | 0 (0) | 39.7 (6.6)
1 | 85 (0.4) | 0.1 (0) | 31.1 (16.8) | 85 (0.4) | 0.1 (0) | 24.8 (13.9) | 85 (0.4) | 0.1 (0) | 17 (10.9) | 85 (0.4) | 0.1 (0) | 16.5 (6.1) | 85 (0.4) | 0.1 (0) | 27 (7.5)
10 | 85 (0.4) | 0 (0) | 13.2 (2.2) | 85 (0.4) | 0 (0) | 9.2 (4.5) | 85 (0.4) | 0 (0) | 17.9 (17.5) | 85 (0.4) | 0 (0) | 10.5 (3.8) | 85 (0.4) | 0 (0) | 18.7 (19.1)
10 | 0 | 79.4 (1.7) | 0.5 (0.5) | 3350.1 (568.7) | 78.6 (0.8) | 0.7 (0.1) | 3606.8 (12.4) | 79.1 (0.8) | 0.6 (0.2) | 3601 (1.2) | 79.3 (1.4) | 1.2 (0.4) | 3600.2 (0.1) | 79.3 (1.5) | 0.7 (0.5) | 3388.8 (474)
0.01 | 85 (0.4) | 3.3 (0.6) | 3600.7 (0.4) | 85 (0.4) | 2.2 (0.6) | 3605.5 (11) | 85 (0.4) | 1.1 (0.1) | 3604.8 (9.7) | 85 (0.4) | 2 (0.2) | 3601.6 (1.6) | 85 (0.4) | 3.3 (0.6) | 3605.5 (11.5)
0.1 | 85 (0.4) | 0.1 (0) | 185.8 (205.3) | 85 (0.4) | 0.1 (0) | 82.3 (11.9) | 85 (0.4) | 0.1 (0) | 74.6 (9.7) | 85 (0.4) | 0.1 (0) | 91.6 (9.3) | 85 (0.4) | 0.1 (0) | 87.1 (12.6)
1 | 85 (0.4) | 0.1 (0) | 48.8 (16.7) | 85 (0.4) | 0 (0) | 48.5 (8.8) | 85 (0.4) | 0 (0) | 41 (13.6) | 85 (0.4) | 0 (0) | 50 (13.4) | 85 (0.4) | 0.1 (0) | 49.3 (14.4)
10 | 85 (0.4) | 0 (0) | 44.6 (9.7) | 85 (0.4) | 0 (0) | 40.5 (6.6) | 85 (0.4) | 0 (0) | 38.9 (6.2) | 85 (0.4) | 0 (0) | 42.8 (6.8) | 85 (0.4) | 0 (0) | 43.5 (10.6)

Table 9: PNN Hyperparamter Tuning Results for 10-covariate data (Athey Model
2) with $\ell_{0}$ Regularization.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 56.8 (2.7) | 0 (0) | 0.1 (0) | 58.4 (5.3) | 0 (0) | 0.1 (0) | 60.2 (4.3) | 0 (0) | 0.1 (0) | 56 (6) | 0 (0) | 0.1 (0) | 61.5 (10.6) | 0 (0) | 0.5 (1)
0.01 | 91.1 (6.3) | 1.3 (1.1) | 2277.4 (1821.7) | 95.8 (0.2) | 0 (0) | 129.5 (162.8) | 95.8 (0.2) | 0.2 (0.3) | 726.6 (1606.6) | 85.3 (10.9) | 1.7 (1.6) | 2200.1 (1917.9) | 85.9 (11.1) | 1.7 (1.6) | 2356.5 (1725.1)
0.1 | 95.8 (0.2) | 0 (0) | 5 (4.2) | 95.8 (0.2) | 0 (0) | 1.9 (0.1) | 95.8 (0.2) | 0 (0) | 1.9 (0.6) | 95.8 (0.2) | 0.8 (1.6) | 726.3 (1606.6) | 95.8 (0.2) | 0.2 (0.3) | 723.2 (1608.3)
1 | 95.8 (0.2) | 0.1 (0) | 1.4 (0.9) | 95.8 (0.2) | 0 (0) | 0.6 (0.1) | 95.8 (0.2) | 0 (0) | 0.5 (0.1) | 95.8 (0.2) | 0 (0) | 3.1 (2.5) | 95.8 (0.2) | 0.1 (0) | 2.6 (2)
10 | 95.8 (0.2) | 0 (0) | 0.5 (0) | 95.8 (0.2) | 0 (0) | 0.6 (0.1) | 95.8 (0.2) | 0 (0) | 0.5 (0.1) | 95.8 (0.2) | 0 (0) | 0.7 (0.3) | 95.8 (0.2) | 0 (0) | 0.7 (0.3)
10 | 0 | 54.1 (4.2) | 0 (0) | 0.3 (0) | 55.4 (4) | 0 (0) | 0.3 (0) | 55 (3.9) | 0 (0) | 0.3 (0) | 48.3 (6.7) | 0 (0) | 0.3 (0) | 56.8 (13) | 0 (0) | 0.7 (1)
0.01 | 92.1 (4.9) | 1.4 (0.9) | 3600.2 (0.1) | 95.8 (0.2) | 0.4 (0.3) | 3002.4 (1336.6) | 95.8 (0.2) | 0.4 (0.4) | 2911.3 (1540.3) | 84.7 (11.3) | 1.6 (0.9) | 3600.2 (0.1) | 86.1 (11.1) | 1.7 (0.9) | 3600.1 (0)
0.1 | 95.8 (0.2) | 0.1 (0) | 11.8 (6.1) | 95.8 (0.2) | 0 (0) | 5.9 (1.8) | 95.8 (0.2) | 0.1 (0) | 6.1 (2.4) | 95.8 (0.2) | 0.4 (0.8) | 1084.6 (1513.9) | 95.8 (0.2) | 0.2 (0.4) | 841.3 (1550.7)
1 | 95.8 (0.2) | 0 (0) | 3.5 (2.1) | 95.8 (0.2) | 0 (0) | 4.2 (2.3) | 95.8 (0.2) | 0 (0) | 5.5 (4.3) | 95.8 (0.2) | 0.1 (0) | 6.5 (3.9) | 95.8 (0.2) | 0 (0) | 6.6 (2.6)
10 | 95.8 (0.2) | 0 (0) | 3.3 (1.5) | 95.8 (0.2) | 0 (0) | 3.6 (1.2) | 95.8 (0.2) | 0 (0) | 3.6 (1.9) | 95.8 (0.2) | 0 (0) | 5.7 (3.2) | 95.8 (0.2) | 0 (0) | 4.6 (2.3)
500 | 3 | 0 | 86.4 (3) | 0.2 (0.1) | 1956.4 (1577.5) | 87.3 (1.6) | 0.1 (0.1) | 1801.7 (1687.7) | 89.3 (2.2) | 0.1 (0) | 55.7 (40.1) | 86.8 (1.6) | 0.1 (0) | 1285.5 (1552.7) | 87.8 (2.3) | 0.2 (0.2) | 1934.5 (1556.5)
0.01 | 95.7 (0.1) | 1.4 (0.4) | 3600.4 (0.5) | 95.7 (0.1) | 0.1 (0) | 311.1 (280.7) | 95.7 (0.1) | 0.1 (0) | 23.8 (10.9) | 95.7 (0.1) | 0 (0) | 653.8 (775.3) | 95.7 (0.1) | 1.2 (0.5) | 3600.3 (0.4)
0.1 | 95.7 (0.1) | 0.1 (0) | 40.4 (4.8) | 95.7 (0.1) | 0.1 (0) | 34.6 (6.2) | 95.7 (0.1) | 0.1 (0) | 17.1 (6.7) | 95.7 (0.1) | 0.1 (0) | 27.3 (13.1) | 95.7 (0.1) | 0 (0) | 52.1 (11.4)
1 | 95.7 (0.1) | 0.1 (0) | 20.3 (20.4) | 95.7 (0.1) | 0 (0) | 24.9 (14.7) | 95.7 (0.1) | 0 (0) | 13.6 (16.1) | 95.7 (0.1) | 0 (0) | 8.3 (3.8) | 95.7 (0.1) | 0.1 (0) | 11.6 (13.4)
10 | 95.7 (0.1) | 0 (0) | 7.3 (4.1) | 95.7 (0.1) | 0 (0) | 9.5 (3.8) | 95.7 (0.1) | 0 (0) | 14.2 (8.3) | 95.7 (0.1) | 0 (0) | 17 (15.8) | 95.7 (0.1) | 0 (0) | 7 (3.9)
10 | 0 | 79.7 (2) | 0 (0) | 90.8 (104.4) | 79 (2.7) | 0 (0) | 86.2 (73.4) | 84.1 (1) | 0 (0) | 64.2 (68) | 80.6 (4.1) | 0 (0) | 74.9 (58.1) | 81 (3.9) | 0 (0) | 475.5 (820.1)
0.01 | 95.7 (0.1) | 1.4 (0.2) | 3600.5 (0.4) | 95.7 (0.1) | 0.5 (0.1) | 3600.6 (0.5) | 95.7 (0.1) | 0.1 (0) | 1529.1 (1891.5) | 95.7 (0.1) | 0.5 (0.1) | 3603.9 (7.2) | 95.7 (0.1) | 1.3 (0.3) | 3600.5 (0.5)
0.1 | 95.7 (0.1) | 0 (0) | 109.7 (24.9) | 95.7 (0.1) | 0.1 (0) | 91.4 (46.9) | 95.7 (0.1) | 0 (0) | 49.1 (24.7) | 95.7 (0.1) | 0.1 (0) | 63.4 (23.6) | 95.7 (0.1) | 0 (0) | 118.9 (34.4)
1 | 95.7 (0.1) | 0 (0) | 62.9 (35.6) | 95.7 (0.1) | 0 (0) | 57.2 (20.8) | 95.7 (0.1) | 0 (0) | 72 (46.2) | 95.7 (0.1) | 0 (0) | 68.8 (30.6) | 95.7 (0.1) | 0 (0) | 72.8 (30.6)
10 | 95.7 (0.1) | 0 (0) | 47.8 (15.8) | 95.7 (0.1) | 0 (0) | 48.1 (23.1) | 95.7 (0.1) | 0 (0) | 44.7 (24.1) | 95.7 (0.1) | 0 (0) | 40.6 (15.7) | 95.7 (0.1) | 0 (0) | 65 (32)

Table 10: PNN Hyperparamter Tuning Results for 20-covariate data (Athey Model
3) with $\ell_{0}$ Regularization.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 87.9 (2.5) | 0.7 (0.9) | 1972.2 (1533.3) | 86.8 (1.6) | 2.4 (1.7) | 2881.4 (1609.1) | 86.2 (2.1) | 1.5 (1.4) | 2478.2 (1563.8) | 86.4 (4.3) | 2.4 (1.9) | 3446 (345.1) | 87.2 (1.8) | 1.6 (3.5) | 1855.4 (1167.9)
0.01 | 87.8 (2.1) | 5 (1.1) | 3600.3 (0.2) | 87.4 (2.1) | 6.2 (3) | 3600.3 (0.1) | 87.5 (2.9) | 5.2 (1.7) | 3600.4 (0.3) | 87.4 (2.2) | 6 (2.3) | 3600.6 (0.2) | 87.6 (2.2) | 5.5 (4.1) | 3600.3 (0.2)
0.1 | 88.1 (1.7) | 9.8 (3.9) | 3600.2 (0.1) | 88.8 (1.9) | 8.2 (5.5) | 3215.8 (859.7) | 88.5 (2) | 4.7 (6.6) | 2306.2 (1467.1) | 88.8 (2.1) | 8.6 (5.4) | 3161.6 (981.3) | 88.4 (2) | 8 (8.9) | 3046.5 (780.9)
1 | 87.1 (4.1) | 9.1 (14.7) | 3030.6 (749.7) | 87.1 (4.1) | 50 (30.1) | 2901.2 (1563.1) | 87.1 (4.1) | 12.5 (27.8) | 2976.9 (560.9) | 86.1 (4.2) | 27 (26) | 3225.2 (837.2) | 87.1 (4.1) | 23.1 (36.3) | 2976.1 (758.7)
10 | 49.6 (0.4) | 0 (0) | 11.3 (5.5) | 49.5 (0.3) | 0 (0) | 9.5 (4.9) | 49.6 (0.4) | 0 (0) | 9.4 (5.8) | 49.5 (0.3) | 0 (0) | 5.2 (0.7) | 50.4 (0.4) | 0 (0) | 11.4 (6.5)
10 | 0 | 85.9 (0.9) | 0 (0) | 104.3 (101.1) | 86 (2.7) | 0 (0.1) | 910.7 (1534.6) | 85.6 (1.9) | 0 (0) | 135.3 (121) | 83.9 (2.7) | 0 (0) | 319.2 (348.5) | 87.1 (2.4) | 0 (0) | 567.3 (1106.7)
0.01 | 87.6 (2.5) | 5.3 (0.8) | 3600.1 (0) | 88.4 (2) | 6.6 (2.7) | 3600.1 (0.1) | 88.3 (2.5) | 5.4 (1.7) | 3600.1 (0.1) | 87.4 (2.2) | 6.6 (2.3) | 3600.1 (0) | 86.7 (3.9) | 4.9 (3) | 3600.1 (0)
0.1 | 88.1 (1.7) | 15 (3.2) | 3600.1 (0) | 88.8 (2) | 14.1 (5.6) | 3600.2 (0.2) | 88.5 (2) | 11.6 (4.3) | 3600.1 (0.1) | 88.8 (2.1) | 14.8 (5.1) | 3600.1 (0) | 88.8 (1.1) | 14.7 (7.3) | 3600.1 (0)
1 | 87.1 (4.1) | 118.4 (23.6) | 3600.1 (0.1) | 87.1 (4.1) | 136.2 (34) | 3600.1 (0) | 84.8 (6.1) | 116.2 (37.5) | 3600.1 (0.1) | 87.3 (4.3) | 123.1 (35.7) | 3600.1 (0) | 87.1 (4.1) | 113.6 (34.4) | 3600.1 (0)
10 | 49.6 (0.4) | 0 (0) | 18 (18.4) | 49.5 (0.3) | 0 (0) | 10.7 (3.9) | 49.6 (0.4) | 0 (0) | 9.7 (3.5) | 49.5 (0.3) | 0 (0) | 11.4 (8.9) | 50.4 (0.4) | 0 (0) | 10.7 (2.5)
500 | 3 | 0 | 90.4 (0.4) | 9.5 (1.5) | 3600.4 (0.3) | 90.9 (0.4) | 7.1 (1.1) | 3600.7 (0.5) | 90.5 (0.6) | 5.6 (1) | 3600.3 (0.3) | 90.2 (0.8) | 7.1 (2.1) | 3600.1 (0) | 90.3 (0.9) | 12.9 (2.3) | 3600.2 (0)
0.01 | 90.4 (0.3) | 13.2 (1.6) | 3600.2 (0.3) | 90.7 (0.5) | 10.1 (1.4) | 3600.2 (0.1) | 90.3 (0.9) | 8.6 (1.2) | 3600.6 (0.5) | 90.8 (0.4) | 11.2 (2.6) | 3600.2 (0.1) | 90.6 (0.6) | 18.2 (3.3) | 3600.2 (0.1)
0.1 | 90 (1.3) | 30.1 (2.8) | 3600.3 (0.4) | 90 (1.3) | 26.4 (2.3) | 3600.2 (0.1) | 90 (1.3) | 24.6 (2.2) | 3600.1 (0.1) | 90.2 (0.8) | 28.1 (3.7) | 3600.6 (1.1) | 90.4 (0.8) | 37.4 (6.7) | 3600.1 (0.1)
1 | 50.3 (0.6) | 1865.3 (1013.4) | 3600.2 (0.1) | 50.4 (0.5) | 2978.8 (2815.2) | 3600.2 (0.1) | 50.1 (0.6) | 2162.1 (1450.9) | 3600.2 (0.1) | 50.3 (0.6) | 1367.5 (606.6) | 3600.2 (0.1) | 50.1 (0.6) | 1869.8 (934.7) | 3600.2 (0.1)
10 | 50.3 (0.6) | 0 (0) | 400.2 (421.9) | 50.4 (0.5) | 0 (0) | 192.7 (150.4) | 50.1 (0.6) | 0 (0) | 196.5 (160.3) | 50.3 (0.6) | 0 (0) | 165.6 (109.3) | 50.1 (0.6) | 0 (0) | 240 (106.5)
10 | 0 | 89.4 (0.8) | 6.9 (1.6) | 3600.5 (0.3) | 89.9 (0.6) | 6.4 (0.9) | 3600.4 (0.2) | 90.1 (0.5) | 5.1 (0.9) | 3600.4 (0.3) | 90.1 (0.7) | 6.6 (1.9) | 3600.4 (0.2) | 89.5 (0.6) | 9.8 (2.5) | 3600.9 (0.8)
0.01 | 90.4 (0.3) | 13.3 (1.6) | 3600.4 (0.1) | 90.7 (0.4) | 10.2 (1.3) | 3600.4 (0.1) | 90.8 (0.5) | 8.9 (1.4) | 3600.3 (0.1) | 90.8 (0.4) | 11.3 (2.6) | 3600.6 (0.8) | 90 (1.5) | 20.3 (5.6) | 3601.5 (2.3)
0.1 | 90.1 (1.2) | 32.2 (3.4) | 3601.1 (1.5) | 90 (1.3) | 27.5 (2.2) | 3600.5 (0.3) | 89.9 (1.3) | 26.8 (1.9) | 3603.1 (3.6) | 90.1 (0.9) | 29.5 (3.9) | 3600.4 (0.1) | 89.4 (2.2) | 41.3 (8.8) | 3601 (1)
1 | 50.3 (0.6) | 2119.1 (1162) | 3615.4 (20.6) | 50.4 (0.5) | 3306.9 (3063.4) | 3606 (12.3) | 50.1 (0.6) | 2411.4 (1580.5) | 3608.1 (17.4) | 50.3 (0.6) | 1536.7 (659.4) | 3600.4 (0.2) | 50.1 (0.6) | 2086 (990.5) | 3600.6 (0.4)
10 | 50.3 (0.6) | 0 (0) | 178.4 (76.4) | 50.4 (0.5) | 0 (0) | 146.4 (28.3) | 50.1 (0.6) | 0 (0) | 158.3 (15.6) | 50.3 (0.6) | 0 (0) | 222.1 (131.4) | 50.1 (0.6) | 0 (0) | 181.9 (81.4)

Table 11: PNN Hyperparamter Tuning Results for 2-covariate data (Athey Model
1) with $\ell_{1}$ Regularization.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 79.2 (4.4) | 0 (0) | 25.7 (47.4) | 76.9 (2.7) | 0 (0) | 12 (7.7) | 76.5 (3.7) | 0 (0) | 6.7 (2.2) | 76.5 (3.7) | 0 (0) | 5.9 (2.2) | 78.3 (2) | 0 (0) | 4.4 (4.1)
0.01 | 80.2 (2.9) | 0.3 (0.1) | 3600.3 (0.2) | 77.7 (3.9) | 0.4 (0.2) | 3600.5 (0.5) | 79.8 (1.7) | 0.3 (0.1) | 3600.4 (0.3) | 80.5 (1.5) | 0.3 (0.1) | 3600.4 (0.3) | 80 (1.2) | 0.2 (0.1) | 3600.3 (0)
0.1 | 80.5 (3.3) | 2.1 (0.6) | 3600.3 (0.1) | 80.8 (3.6) | 2.5 (0.9) | 3600.2 (0.1) | 82.9 (1.9) | 2.1 (0.7) | 3600.2 (0.1) | 82.3 (1.2) | 2 (0.8) | 3600.3 (0.2) | 82.2 (1.2) | 1.8 (0.7) | 3600.3 (0.1)
1 | 85.1 (0.5) | 0.6 (1.4) | 964.3 (1484.4) | 85.1 (0.5) | 1.4 (3.2) | 1589.4 (1379.4) | 85.1 (0.5) | 0.5 (1.1) | 744.2 (1596.6) | 85.1 (0.5) | 0 (0) | 790.1 (1573.4) | 85.1 (0.5) | 1 (1.6) | 1517.8 (1902.1)
10 | 85.1 (0.5) | 0 (0) | 1.4 (1) | 85.1 (0.5) | 0 (0) | 2 (1.2) | 85.1 (0.5) | 0 (0) | 0.9 (0.8) | 85.1 (0.5) | 0 (0) | 1.9 (1.8) | 85.1 (0.5) | 0 (0) | 1.8 (0.8)
10 | 0 | 74.7 (2.7) | 0 (0) | 8.2 (4.9) | 74.2 (2.6) | 0 (0) | 11 (9.2) | 77.8 (2.5) | 0 (0) | 9.2 (5.8) | 76.8 (1.8) | 0 (0) | 9.1 (9.1) | 78.9 (2.2) | 0 (0) | 4.5 (2.5)
0.01 | 80.5 (2.2) | 0.3 (0.1) | 3600.2 (0.2) | 78.4 (4) | 0.4 (0.1) | 3600.1 (0.1) | 78.2 (3) | 0.4 (0.1) | 3600.1 (0.1) | 79.4 (2.6) | 0.3 (0.1) | 3600.2 (0.1) | 79.7 (1.5) | 0.3 (0.1) | 3600.1 (0)
0.1 | 80.5 (3.5) | 2.3 (0.6) | 3600.1 (0) | 81.4 (2.4) | 2.8 (1) | 3600.1 (0.1) | 82.6 (2.2) | 2.3 (0.7) | 3600.1 (0) | 82.3 (1.2) | 2.2 (0.8) | 3600.2 (0.1) | 81.9 (0.9) | 2.1 (0.7) | 3600.1 (0.1)
1 | 85.1 (0.5) | 2.8 (2.5) | 3001.8 (1337.7) | 85.1 (0.5) | 4.1 (3.6) | 3013 (1312.8) | 85.1 (0.5) | 0.6 (1.3) | 2705.7 (1276.9) | 85.1 (0.5) | 1.5 (1.8) | 2214.7 (1898.2) | 85.1 (0.5) | 2.6 (3.5) | 3084.2 (1153.5)
10 | 85.1 (0.5) | 0 (0) | 31.7 (29.8) | 85.1 (0.5) | 0 (0) | 19.8 (16.3) | 85.1 (0.5) | 0 (0) | 8.6 (3.2) | 85.1 (0.5) | 0 (0) | 13 (8.8) | 85.1 (0.5) | 0 (0) | 14.1 (6.4)
500 | 3 | 0 | 81.3 (1.5) | 2.4 (1.5) | 3600.1 (0) | 82.8 (0.6) | 2.3 (0.4) | 3600.2 (0.1) | 83 (0.7) | 1.5 (0.3) | 3600.3 (0.2) | 82.4 (0.3) | 2.6 (0.4) | 3600.7 (0.5) | 80.5 (2.9) | 2.7 (0.6) | 3600.2 (0.1)
0.01 | 82.9 (0.6) | 3.7 (1.6) | 3600.2 (0.1) | 83.6 (0.3) | 3.3 (0.3) | 3600.1 (0) | 84.1 (0.5) | 2.2 (0.3) | 3600.2 (0.1) | 83.9 (0.5) | 3.6 (0.4) | 3600.2 (0.1) | 83.1 (0.7) | 4.4 (0.5) | 3600.1 (0.1)
0.1 | 84.8 (0.6) | 7.2 (2) | 3600.2 (0.1) | 85 (0.4) | 5.2 (1) | 3600.4 (0.2) | 85 (0.4) | 2.9 (0.2) | 3600.1 (0.1) | 85 (0.4) | 5.3 (0.8) | 3600.7 (0.9) | 84.5 (0.8) | 7.9 (1) | 3600.3 (0.3)
1 | 85 (0.4) | 5.8 (1.7) | 3600.1 (0) | 85 (0.4) | 2.7 (1.4) | 3600.2 (0) | 85 (0.4) | 0.2 (0.3) | 2153.5 (1012) | 85 (0.4) | 2.3 (0.7) | 3600.4 (0.5) | 85 (0.4) | 6 (1.3) | 3600.2 (0.1)
10 | 85 (0.4) | 0 (0) | 33.7 (17.4) | 85 (0.4) | 0 (0) | 32 (9.8) | 85 (0.4) | 0 (0) | 25.8 (8.3) | 85 (0.4) | 0 (0) | 36.6 (24.7) | 85 (0.4) | 0 (0) | 54.5 (27.1)
10 | 0 | 78.7 (2.9) | 0.5 (0.4) | 3271 (737.5) | 78.1 (1.1) | 0.7 (0.2) | 3600.6 (0.6) | 79 (1.9) | 0.6 (0.2) | 3601.1 (1.9) | 78.7 (1) | 1.2 (0.2) | 3601 (0.9) | 78.8 (3.4) | 0.5 (0.3) | 3098.2 (1122.8)
0.01 | 82 (1.4) | 2.7 (0.7) | 3600.9 (0.5) | 83.1 (0.7) | 2.9 (0.2) | 3602.2 (4) | 83.8 (0.7) | 2.3 (0.2) | 3600.4 (0.3) | 83 (0.9) | 3.5 (0.3) | 3600.7 (0.6) | 81.6 (0.9) | 3.5 (0.9) | 3601.1 (0.8)
0.1 | 84.8 (0.5) | 7.7 (2.1) | 3601.5 (2.4) | 85 (0.4) | 5.6 (1.3) | 3600.6 (0.4) | 85 (0.4) | 3 (0.3) | 3600.5 (0.4) | 84.9 (0.5) | 5.3 (0.6) | 3602.8 (5.4) | 84.9 (0.5) | 8.6 (1.4) | 3600.8 (0.6)
1 | 85 (0.4) | 7.6 (1.3) | 3600.2 (0.2) | 85 (0.4) | 4.4 (1.3) | 3600.4 (0.2) | 85 (0.4) | 1.9 (0.4) | 3600.9 (0.6) | 85 (0.4) | 4.2 (0.9) | 3601.6 (2.1) | 85 (0.4) | 7.6 (1.3) | 3604.6 (7.9)
10 | 85 (0.4) | 0 (0) | 269.1 (98.1) | 85 (0.4) | 0 (0) | 201.3 (61) | 85 (0.4) | 0 (0) | 120.6 (60.1) | 85 (0.4) | 0 (0) | 174.2 (66.4) | 85 (0.4) | 0 (0) | 257.9 (41.3)

Table 12: PNN Hyperparameter Tuning Results for 10-covariate data (Athey Model
2) with $\ell_{1}$ Regularization.

Train Data Length | Width | Hyper parameter | Probability of Correct Assignments
---|---|---|---
0.1 | 0.25 | 0.5 | 0.75 | 0.9
OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s) | OOSP (%) | Gap (%) | Runtime (s)
100 | 3 | 0 | 56.8 (2.7) | 0 (0) | 0.1 (0) | 58.4 (5.3) | 0 (0) | 0.1 (0) | 60.2 (4.3) | 0 (0) | 0.1 (0) | 56 (6) | 0 (0) | 0.1 (0) | 61.5 (10.6) | 0 (0) | 0.5 (0.9)
0.01 | 84.9 (7.6) | 0.1 (0) | 0.1 (0.1) | 89.2 (5.4) | 0.1 (0) | 0.1 (0.2) | 92.6 (1.9) | 0.1 (0) | 4.7 (6.4) | 83.6 (9.5) | 0.1 (0) | 8.5 (18.9) | 83.1 (8.9) | 0.1 (0) | 734.6 (1602.1)
0.1 | 87.8 (6.2) | 0.3 (0.1) | 3600.1 (0) | 93.6 (0.7) | 0.2 (0.2) | 2862.8 (1022) | 93.3 (1.1) | 0.2 (0.2) | 2287.7 (1802.4) | 84.8 (8.1) | 0.4 (0.2) | 3470.3 (290.3) | 85.1 (6.9) | 0.6 (0.3) | 3600.1 (0.1)
1 | 93.6 (1.9) | 1.8 (1.5) | 2891.1 (1585.4) | 95.8 (0.2) | 0 (0) | 506.7 (762.7) | 95.8 (0.2) | 0.4 (0.8) | 798.9 (1568.6) | 87.6 (9.3) | 1.6 (1.4) | 2355 (1719.4) | 88.3 (8.9) | 2.1 (1.7) | 2886.5 (1595.7)
10 | 95.8 (0.2) | 0 (0) | 3.6 (2.8) | 95.8 (0.2) | 0 (0) | 0.7 (0.3) | 95.8 (0.2) | 0 (0) | 1 (1) | 93 (6.1) | 4.3 (9.6) | 736.2 (1601.1) | 95.8 (0.2) | 3.8 (8.5) | 724.1 (1607.7)
10 | 0 | 54.1 (4.2) | 0 (0) | 0.3 (0) | 55.4 (4) | 0 (0) | 0.3 (0) | 55 (3.9) | 0 (0) | 0.3 (0) | 48.3 (6.7) | 0 (0) | 0.3 (0) | 56.8 (13) | 0 (0) | 0.9 (1.4)
0.01 | 84.9 (7.6) | 0.1 (0) | 0.6 (0.7) | 89.2 (5.4) | 0.1 (0) | 0.6 (0.7) | 92.8 (1.4) | 0.1 (0) | 20.5 (33.9) | 83.6 (9.5) | 0.1 (0) | 57.2 (127.1) | 83.8 (8.7) | 0.1 (0) | 836.6 (1564.8)
0.1 | 87.8 (6.2) | 0.5 (0.1) | 3600.1 (0.1) | 93.6 (0.7) | 0.4 (0.2) | 3600.1 (0) | 93.3 (1.1) | 0.4 (0.2) | 3600.1 (0) | 85.5 (8.7) | 0.5 (0.2) | 3600.2 (0.1) | 84.9 (6.7) | 0.7 (0.3) | 3600.2 (0.1)
1 | 93.6 (1.9) | 2.7 (1.8) | 3429.7 (381) | 95.8 (0.2) | 0.8 (0.6) | 3241 (803) | 95.8 (0.2) | 0.7 (1.2) | 2198.3 (1634.8) | 87.6 (9.3) | 2.3 (1.6) | 3600.2 (0.1) | 88.3 (8.9) | 3.3 (1.9) | 3600.1 (0)
10 | 95.8 (0.2) | 0 (0) | 140.3 (169.9) | 95.8 (0.2) | 0 (0) | 15 (7.6) | 95.8 (0.2) | 0 (0) | 13.3 (7) | 95.8 (0.2) | 8.4 (18.7) | 1266.6 (1463.8) | 95.8 (0.2) | 7 (15.7) | 884.1 (1530.8)
500 | 3 | 0 | 86.4 (3) | 0.2 (0.1) | 1980.4 (1568.9) | 87.2 (1.4) | 0.1 (0.1) | 1819.6 (1676.9) | 89.3 (2.2) | 0.1 (0) | 59.3 (42.2) | 86.8 (1.6) | 0.1 (0) | 1333.6 (1566) | 87.8 (2.3) | 0.2 (0.2) | 1976.9 (1524.1)
0.01 | 90.3 (0.8) | 0.7 (0.2) | 3600.4 (0.5) | 91.9 (0.5) | 0.6 (0.1) | 3600.4 (0.4) | 93.8 (0.9) | 0.2 (0) | 3600.1 (0) | 92.6 (0.7) | 0.4 (0.1) | 3600.1 (0.1) | 89.6 (1) | 0.7 (0.2) | 3600.1 (0.1)
0.1 | 94.4 (0.6) | 2.6 (0.5) | 3600.1 (0.1) | 95.7 (0.1) | 1.4 (0.2) | 3600.2 (0) | 95.7 (0.1) | 0.5 (0.2) | 3600.5 (0.6) | 95.7 (0.1) | 1.3 (0.3) | 3600.4 (0.6) | 94.1 (0.3) | 2.4 (0.3) | 3600.2 (0.1)
1 | 95.7 (0.1) | 2.4 (0.5) | 3600.2 (0.1) | 95.7 (0.1) | 0.1 (0.1) | 1995.3 (1388.4) | 95.7 (0.1) | 0.1 (0) | 47.7 (19.7) | 95.7 (0.1) | 0 (0) | 1416.9 (1324.2) | 95.7 (0.1) | 2.3 (0.7) | 3600.2 (0.1)
10 | 95.7 (0.1) | 0 (0) | 66.1 (29.8) | 95.7 (0.1) | 0 (0) | 78.4 (18.3) | 95.7 (0.1) | 0 (0) | 37.1 (22.3) | 95.7 (0.1) | 0 (0) | 65.5 (33.3) | 95.7 (0.1) | 0 (0) | 98.4 (53.5)
10 | 0 | 79.7 (2) | 0 (0) | 105.3 (120.5) | 79 (2.7) | 0 (0) | 99.8 (85.6) | 84.1 (1) | 0 (0) | 75.7 (79.9) | 80.6 (4.1) | 0 (0) | 84.4 (64.5) | 81 (3.9) | 0 (0) | 555.6 (958.4)
0.01 | 92.8 (1.2) | 0.4 (0) | 3601.5 (1.8) | 92.5 (1.1) | 0.5 (0.1) | 3602.8 (4.1) | 94.1 (0.6) | 0.3 (0) | 3603 (5.1) | 92.7 (1.1) | 0.4 (0.1) | 3600.6 (0.3) | 92 (0.7) | 0.4 (0) | 3600.8 (0.9)
0.1 | 94.5 (0.6) | 2.7 (0.5) | 3601.5 (2) | 95.7 (0.1) | 1.4 (0.2) | 3602.1 (1.9) | 95.7 (0.1) | 0.6 (0.1) | 3601.4 (2.1) | 95.7 (0.1) | 1.3 (0.3) | 3601.5 (2.7) | 94.4 (0.6) | 2.6 (0.4) | 3608.6 (10.1)
1 | 95.7 (0.1) | 3.4 (0.4) | 3600.4 (0.1) | 95.7 (0.1) | 1.2 (0.2) | 3600.6 (0.1) | 95.7 (0.1) | 0.1 (0.1) | 2094.5 (904.1) | 95.7 (0.1) | 1 (0.3) | 3600.5 (0.1) | 95.7 (0.1) | 3.4 (0.7) | 3601.3 (0.9)
10 | 95.7 (0.1) | 0 (0) | 471.4 (190.6) | 95.7 (0.1) | 0 (0) | 193.5 (72.4) | 95.7 (0.1) | 0 (0) | 166.7 (55) | 95.7 (0.1) | 0 (0) | 232.3 (113.5) | 95.7 (0.1) | 0 (0) | 540.7 (419.4)

Table 13: PNN Hyperparamter Tuning Results for 20-covariate data (Athey Model
3) with $\ell_{1}$ Regularization.

####  D.2.2 Personalized Postpartum Hypertension Treatment

Please see Table 14.

Experiment |  Width |  Hyper Parameter |  Gap (%) |  Runtime (s) |  Average Outcome |  95% Confidence Interval |  90% Confidence Interval
---|---|---|---|---|---|---|---
1 | 3 | 0 | 0.01 | 3600.31 | 148.41 | [147.08, 149.74] | [147.08, 149.74]
0.01 | 0.01 | 3600.31 | 148.5 | [147.14, 149.87] | [147.14, 149.87]
0.1 | 0.03 | 3600.42 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
1 | 0.01 | 3041.54 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
10 | 0 | 5.49 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
10 | 0 | 0 | 2333.51 | 145.76 | [143.71, 147.81] | [143.71, 147.81]
0.01 | 0.01 | 3600.36 | 149.02 | [147.79, 150.26] | [147.79, 150.26]
0.1 | 0.04 | 3600.38 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
1 | 0.03 | 3600.37 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
10 | 0 | 34.95 | 146.43 | [145.2, 147.66] | [145.2, 147.66]
2 | 3 | 0 | 0.03 | 3600.14 | 148.1 | [146.32, 149.88] | [146.32, 149.88]
0.01 | 0.03 | 3601.03 | 149.46 | [147.71, 151.22] | [147.71, 151.22]
0.1 | 0.05 | 3602.06 | 149.77 | [147.67, 151.87] | [147.67, 151.87]
1 | 0.07 | 3600.48 | 157.07 | [146.98, 167.17] | [146.98, 167.17]
10 | 0 | 102.39 | 148.11 | [146.26, 149.97] | [146.26, 149.97]
10 | 0 | 0.01 | 3600.97 | 147.3 | [144.48, 150.11] | [144.48, 150.11]
0.01 | 0.02 | 3600.4 | 158.12 | [147.85, 168.39] | [147.85, 168.39]
0.1 | 0.07 | 3600.3 | 154.68 | [144.66, 164.7] | [144.66, 164.7]
1 | 0.09 | 3600.37 | 156.82 | [146.72, 166.92] | [146.72, 166.92]
10 | 0 | 644.74 | 150.68 | [148.66, 152.69] | [148.66, 152.69]
3 | 3 | 0 | 0.05 | 3600.11 | 148.83 | [147.17, 150.49] | [147.17, 150.49]
0.01 | 0.06 | 3600.17 | 146.89 | [145.93, 147.85] | [145.93, 147.85]
0.1 | 0.07 | 3600.61 | 149.1 | [146.97, 151.24] | [146.97, 151.24]
1 | 0.09 | 3600.16 | 146.05 | [144.07, 148.03] | [144.07, 148.03]
10 | 0 | 1408.52 | 146.92 | [145.06, 148.79] | [145.06, 148.79]
10 | 0 | 0.06 | 3600.31 | 147.4 | [144.15, 150.64] | [144.15, 150.64]
0.01 | 0.05 | 3600.5 | 157.74 | [153.55, 161.92] | [153.55, 161.92]
0.1 | 0.09 | 3600.34 | 145.38 | [143.65, 147.12] | [143.65, 147.12]
1 | 0.07 | 3601.26 | 146.79 | [145.12, 148.46] | [145.12, 148.46]
10 | 0 | 3233.17 | 146.14 | [144.56, 147.71] | [144.56, 147.71]

Table 14: PNN Hyperparamter Tuning Results for Hypertension Experiments with
$\ell_{0}$ Regularization.

###  D.3 Model Interpretation Results

In this section, we provide the complete results of the model interpretation
analyses.

####  D.3.1 Personalized Postpartum Hypertension Treatments

For the postpartum hypertension experiments, we ranked the features by their
SHAP values for each model. To account for the 10 cross-validation folds,
these rankings were then averaged. Since the prescriptive trees branch on at
most 3 features, those that were not chosen to branch on were given the lowest
ranking. Recall that the prescriptive trees optimize on a binarized feature
space. To fairly compare the PNN and CF models with the tree models, we took
the minimum rank for each binarized feature as the reported rank. Finally, to
account for varied number of features due the binarization, the rankings were
normalized between 0 and 1.

The feature names in Tables 16-18 correspond to the demographic and medical
characteristics Table 15.

Feature Name |  Description
---|---
mode.del |  Mode of delivery: Cesarean/Vaginal
chronicHTN |  Chronic Hypertension: Yes/No
gestHTN |  Gestational Hypertension: Yes/No
bmi.prenatal |  Pre-natal maternal BMI (Continuous)
sipe |  Superimposed Pre-eclampsia: Yes/No
race |  White, Black, Other
ethnicity |  Non-hispanic, Hispanic
feeding.br |  Breast feeding: Yes/No
mom.age |  Maternal age (Discrete)
ethnicity |  Non-hispanic, Hispanic
insurance |  Private, Medicaid, Other, Unknown
tobacco |  Tobacco use: Yes/No
DM.gest |  Gestational Diabetes: Yes/No
gest.age |  Gestational Age (Discrete)
pree |  Pre-eclampsia: Yes/No
pree.w.sf |  Pre-eclampsia with Severe Features: Yes/No
eclampsia |  Eclampsia: Yes/No
hellp |  Hemolysis, Elevated Liver Enzymes, Low Platelet Count: Yes/No

Table 15: Feature names and their descriptions for the postpartum hypertension
experiments.

Model | Architecture | mode.del | chronicHTN | gestHTN | bmi.prenatal | sipe | race | ethnicity | feeding.br | mom.age | DM.pregest | insurance | tobacco | DM.gest | gest.age | pree | pree.w.sf | eclampsia | hellp | dcDayPP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CF | 100 | 0.600 | 0.509 | 0.691 | 0.915 | 0.412 | 0.376 | 0.309 | 0.273 | 0.915 | 0.200 | 0.055 | 0.158 | 0.279 | 0.776 | 0.636 | 0.545 | 0.048 | 0.000 | 0.697
J-PT | 1 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.981
2 | 0.000 | 0.200 | 0.000 | 0.300 | 0.000 | 0.000 | 0.000 | 0.200 | 0.100 | 0.200 | 0.494 | 0.000 | 0.000 | 0.400 | 0.000 | 0.000 | 0.100 | 0.094 | 0.100
B-PT | 1 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
2 | 0.778 | 0.222 | 0.000 | 0.222 | 0.000 | 0.000 | 0.000 | 0.333 | 0.222 | 0.111 | 0.222 | 0.000 | 0.000 | 0.222 | 0.000 | 0.000 | 0.222 | 0.000 | 0.222
K-PT | 1 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
2 | 0.000 | 0.000 | 0.000 | 0.556 | 0.000 | 0.111 | 0.000 | 0.111 | 0.111 | 0.000 | 0.111 | 0.000 | 0.000 | 0.333 | 0.000 | 0.000 | 0.000 | 0.000 | 0.111
PNN | 3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
10 | 0.614 | 0.659 | 0.777 | 0.677 | 0.373 | 0.277 | 0.186 | 0.168 | 0.682 | 0.164 | 0.282 | 0.150 | 0.382 | 0.627 | 0.850 | 0.595 | 0.000 | 0.250 | 0.877

Table 16: Normalized Rankings of Features for Postpartum Hypertension
Experiment 1.

Model | Architecture | mode.del | chronicHTN | gestHTN | bmi.prenatal | sipe | race | ethnicity | feeding.br | mom.age | DM.pregest | insurance | tobacco | DM.gest | gest.age | pree | pree.w.sf | eclampsia | hellp | bpmed
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CF | 100 | 0.474 | 0.684 | 0.605 | 0.874 | 0.605 | 0.426 | 0.168 | 0.342 | 0.932 | 0.153 | 0.247 | 0.142 | 0.405 | 0.868 | 0.616 | 0.689 | 0.147 | 0.000 | 0.621
J-PT | 1 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.300 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.600 | 0.000 | 0.000
2 | 0.096 | 0.000 | 0.000 | 0.265 | 0.100 | 0.100 | 0.590 | 0.194 | 0.100 | 0.100 | 0.400 | 0.100 | 0.000 | 0.200 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
B-PT | 1 | 0.007 | 0.380 | 0.007 | 0.380 | 0.000 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.007 | 0.251
2 | 0.006 | 0.006 | 0.006 | 0.168 | 0.495 | 0.006 | 0.404 | 0.105 | 0.801 | 0.006 | 0.006 | 0.000 | 0.006 | 0.006 | 0.006 | 0.006 | 0.199 | 0.105 | 0.090
K-PT | 1 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.167 | 0.000 | 0.000 | 0.000 | 0.000 | 0.817
2 | 0.780 | 0.000 | 0.000 | 0.900 | 0.000 | 0.000 | 0.000 | 0.049 | 0.000 | 0.000 | 0.090 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.969
PNN | 3 | 0.821 | 0.663 | 0.737 | 0.532 | 0.558 | 0.437 | 0.337 | 0.168 | 0.658 | 0.211 | 0.279 | 0.168 | 0.368 | 0.668 | 0.837 | 0.658 | 0.000 | 0.147 | 0.753
10 | 0.926 | 0.593 | 0.825 | 0.667 | 0.450 | 0.402 | 0.175 | 0.175 | 0.661 | 0.233 | 0.233 | 0.206 | 0.434 | 0.656 | 0.815 | 0.587 | 0.000 | 0.116 | 0.794

Table 17: Normalized Rankings of Features for Postpartum Hypertension
Experiment 2.

Model | Architecture | mode.del | chronicHTN | gestHTN | bmi.prenatal | sipe | race | ethnicity | feeding.br | mom.age | DM.pregest | insurance | tobacco | DM.gest | gest.age | pree | pree.w.sf | eclampsia | hellp
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CF | 100 | 0.500 | 0.772 | 0.617 | 0.889 | 0.544 | 0.494 | 0.178 | 0.333 | 0.906 | 0.167 | 0.256 | 0.078 | 0.439 | 0.867 | 0.622 | 0.656 | 0.183 | 0.000
J-PT | 1 | 0.000 | 0.000 | 0.000 | 0.800 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
2 | 0.386 | 0.492 | 0.100 | 0.200 | 0.000 | 0.100 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.200 | 0.394 | 0.000 | 0.200 | 0.000
B-PT | 1 | 0.784 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
2 | 0.362 | 0.000 | 0.000 | 0.100 | 0.000 | 0.000 | 0.000 | 0.300 | 0.290 | 0.200 | 0.500 | 0.300 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000
K-PT | 1 | 0.980 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
2 | 0.568 | 0.000 | 0.000 | 0.300 | 0.100 | 0.000 | 0.296 | 0.494 | 0.100 | 0.000 | 0.000 | 0.000 | 0.000 | 0.100 | 0.000 | 0.196 | 0.000 | 0.000
PNN | 3 | 0.463 | 0.510 | 0.503 | 0.698 | 0.228 | 0.376 | 0.342 | 0.148 | 0.510 | 0.262 | 0.322 | 0.020 | 0.362 | 0.544 | 0.497 | 0.651 | 0.087 | 0.000
10 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A

Table 18: Normalized Rankings of Features for Postpartum Hypertension
Experiment 3.