Title: Effective Offline Reinforcement Learning via Conservative State Value Estimation. Abstract: Offline RL promises to learn effective policies from static experience datasets without further interaction, which expect to perform well in the online environment. However, it faces up to a major challenge of value over-estimation introduced by the distributional drift between the dataset and the current learned policy, which leads to learning failure in practice. The common approach is to add a penalty term to reward or value estimation in the Bellman iterations, which has given rise to a number of successful algorithms such as CQL. Meanwhile, to avoid extrapolation on unseen states, existing methods focus on conservative Q-function estimation. In this paper, we propose CSVE, a new approach that directly imposes penalty on out-of-distribution states. We prove that for the evaluated policy, our conservative state value estimation satisfies: (1) over the state distribution that samples penalizing states, it lower bounds the true values in expectation, and (2) over the marginal state distribution of data, it is no more than the true values in expectation plus a constant decided by sampling error. Further, we develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states 'around' the dataset, while the actor applies advantage weighted updates to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods (e.g., CQL) and is strongly competitive among recent SOTA methods.