Polymer micelles are promising drug delivery systems for highly hydrophobic photosensitizers in photodynamic therapy (PDT) applications. We previously developed pH-responsive polymer micelles consisting of poly(styrene-co-2-(N,N-dimethylamino)ethyl acrylate)-block-poly(polyethylene glycol monomethyl ether acrylate) (P(St-co-DMAEA)-b-PPEGA) for zinc phthalocyanine (ZnPc) delivery. In this study, poly(butyl-co-2-(N,N-dimethylamino)ethyl acrylates)-block-poly(polyethylene glycol monomethyl ether acrylate) (P(BA-co-DMAEA)-b-PPEGA) was synthesized via reversible addition and fragmentation chain transfer (RAFT) polymerization to explore the role of neutral hydrophobic units in photosensitizer delivery. The composition of DMAEA units in P(BA-co-DMAEA) was adjusted to 0.46, which is comparable to that of P(St-co-DMAEA)-b-PPEGA. The size distribution of the P(BA-co-DMAEA)-b-PPEGA micelles changed when the pH decreased from 7.4 to 5.0, indicating their pH-responsive ability. The photosensitizers, 5,10,15,20-tetrakis(pentafluorophenyl)chlorin (TFPC), 5,10,15,20-tetrakis(pentafluorophenyl)porphyrin (TFPP), protoporphyrin IX (PPIX), and ZnPc were examined as payloads for the P(BA-co-DMAEA)-b-PPEGA micelles. The encapsulation efficiency depended on the nature of the photosensitizer. TFPC-loaded P(BA-co-DMAEA)-b-PPEGA micelles exhibited higher photocytotoxicity than free TFPC in the MNNG-induced mutant of the rat murine RGM-1 gastric epithelial cell line (RGK-1), indicating their superiority for photosensitizer delivery. ZnPc-loaded P(BA-co-DMAEA)-b-PPEGA micelles also exhibited superior photocytotoxicity compared to free ZnPc. However, their photocytotoxicity was lower than that of P(St-co-DMAEA)-b-PPEGA. Therefore, neutral hydrophobic units, as well as pH-responsive units, must be designed for the encapsulation of photosensitizers.